<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 22, 2017 at 3:22 PM, Michal Skrivanek <span dir="ltr"><<a href="mailto:michal.skrivanek@redhat.com" target="_blank">michal.skrivanek@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><br><div><span class="gmail-"><blockquote type="cite"><div>On 22 Feb 2017, at 13:53, Simone Tiraboschi <<a href="mailto:stirabos@redhat.com" target="_blank">stirabos@redhat.com</a>> wrote:</div><br class="gmail-m_4346065651595802519Apple-interchange-newline"><div><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 22, 2017 at 1:33 PM, Simone Tiraboschi <span dir="ltr"><<a href="mailto:stirabos@redhat.com" target="_blank">stirabos@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">When ovirt-ha-agent checks the status of the engine VM we get:<div><pre>2017-02-21 22:21:14,738-0500 ERROR (jsonrpc/2) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'2ccc0ef0-cc31-45b8-8e91-a78f<wbr>a4cad671'} (api:69)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/common/api.py", line 67, in method
ret = func(*args, **kwargs)
File "/usr/share/vdsm/API.py", line 335, in getStats
vm = self.vm
File "/usr/share/vdsm/API.py", line 130, in vm
raise exception.NoSuchVM(vmId=self._<wbr>UUID)
NoSuchVM: Virtual machine does not exist: {'vmId': u'2ccc0ef0-cc31-45b8-8e91-a78f<wbr>a4cad671'}</pre><pre><br></pre><pre>While in ovirt-ha-agent logs we have:<pre><pre>MainThread::INFO::2017-02-21 22:21:18,583::hosted_engine::4<wbr>53::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(start_monitoring) Current state UnknownLocalVmState (score: 3400)</pre><pre>...</pre></pre><pre>MainThread::INFO::2017-02-21 22:21:31,199::state_decorators<wbr>::25::ovirt_hosted_engine_ha.<wbr>agent.hosted_engine.HostedEngi<wbr>ne::(check) Unknown local engine vm status no actions taken</pre></pre>Probably it's a bug or a regression somewhere on master.</div></div></blockquote><div><br></div><div>On ovirt-ha-broker side the detection is based on a strict string match on the error message that is expected to be exactly 'Virtual machine does not exist' to set down status otherwise we set unknown status as in this case:</div><div><a href="https://gerrit.ovirt.org/gitweb?p=ovirt-hosted-engine-ha.git;a=blob;f=ovirt_hosted_engine_ha/broker/submonitors/engine_health.py;h=d633cb860b811e84021221771bf706a9a4ac1d63;hb=refs/heads/master#l54" target="_blank">https://gerrit.ovirt.org/<wbr>gitweb?p=ovirt-hosted-engine-<wbr>ha.git;a=blob;f=ovirt_hosted_<wbr>engine_ha/broker/submonitors/<wbr>engine_health.py;h=<wbr>d633cb860b811e84021221771bf706<wbr>a9a4ac1d63;hb=refs/heads/<wbr>master#l54</a><br></div><div> </div><div>Adding Francesco here to understand if something has recently changed there on vdsm side.</div></div></div></div></div></blockquote><div><br></div></span>That’s not a very robust code handling.</div><div>Yes, the text changed, the vm id was added.</div><div>And yes, it may change again any time I guess</div></div></blockquote><div><br></div><div>I agree, we are going to move to code check: <a href="https://gerrit.ovirt.org/#/c/72891">https://gerrit.ovirt.org/#/c/72891</a></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div><div class="gmail-h5"><div><br></div><div><blockquote type="cite"><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div class="gmail-m_4346065651595802519gmail-h5"><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 22, 2017 at 1:02 PM, Sandro Bonazzola <span dir="ltr"><<a href="mailto:sbonazzo@redhat.com" target="_blank">sbonazzo@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Adding Lev</div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="gmail-m_4346065651595802519gmail-m_3795450909556566802gmail-h5">On Wed, Feb 22, 2017 at 12:59 PM, Sahina Bose <span dir="ltr"><<a href="mailto:sabose@redhat.com" target="_blank">sabose@redhat.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-m_4346065651595802519gmail-m_3795450909556566802gmail-h5"><div dir="ltr"><pre>Hi all,<br><br>On the HC setup, the HE VM is not restarted.<br>The agent.log has <br>MainThread::INFO::2017-02-21 22:09:58,022::state_machine::1<wbr>69::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(refresh) Global metadata: {}
MainThread::INFO::2017-02-21 22:09:58,023::state_machine::1<wbr>77::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(refresh) Local (id 1): {'engine-health': {'reason': 'failed to getVmStats', 'health': 'unknown', 'vm': 'unknown', 'detail': 'unknown'}, 'bridge': True, 'mem-free': 4079.0, 'maintenance': False, 'cpu-load': 0.0491, 'gateway': True}<br>...<br>MainThread::INFO::2017-02-21 22:10:29,219::state_decorators<wbr>::25::ovirt_hosted_engine_ha.a<wbr>gent.hosted_engine.HostedEngin<wbr>e::(check) Unknown local engine vm status no actions taken
MainThread::INFO::2017-02-21 22:10:29,219::brokerlink::111:<wbr>:<a href="http://ovirt_hosted_engine_ha.lib.br/" target="_blank">ovirt_hosted_engine_ha.lib.br</a><wbr>okerlink.BrokerLink::(notify) Trying: notify time=1487733029.22 type=state_transition detail=ReinitializeFSM-Unknown<wbr>LocalVmState hostname='lago-hc-basic-suite-<wbr>master-host0'
MainThread::INFO::2017-02-21 22:10:29,317::brokerlink::121:<wbr>:<a href="http://ovirt_hosted_engine_ha.lib.br/" target="_blank">ovirt_hosted_engine_ha.lib.br</a><wbr>okerlink.BrokerLink::(notify) Success, was notification of state_transition (ReinitializeFSM-UnknownLocalV<wbr>mState) sent? ignored<br></pre><pre>and the vdsm.log <br><br>2017-02-21 22:09:11,962-0500 INFO (libvirt/events) [virt.vm] (vmId='2ccc0ef0-cc31-45b8-8e91<wbr>-a78fa4cad671') Changed state to Down: User shut down from within the guest (code=7) (vm:1269)
2017-02-21 22:09:11,962-0500 INFO (libvirt/events) [virt.vm] (vmId='2ccc0ef0-cc31-45b8-8e91<wbr>-a78fa4cad671') Stopping connection (guestagent:429)<br><br>2017-02-21 22:09:29,727-0500 ERROR (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'2ccc0ef0-cc31-45b8-8e91-a78f<wbr>a4cad671'} (api:69)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/common/api.py", line 67, in method
ret = func(*args, **kwargs)
File "/usr/share/vdsm/API.py", line 335, in getStats
vm = self.vm
File "/usr/share/vdsm/API.py", line 130, in vm
raise exception.NoSuchVM(vmId=self._<wbr>UUID)
NoSuchVM: Virtual machine does not exist: {'vmId': u'2ccc0ef0-cc31-45b8-8e91-a78f<wbr>a4cad671'}<br><br><br></pre><pre>What should I be looking for to identify the issue?<br><br></pre><pre>The logs are at <a href="http://jenkins.ovirt.org/job/ovirt_master_hc-system-tests/lastCompletedBuild/artifact/exported-artifacts/test_logs/hc-basic-suite-master/post-002_bootstrap.py/lago-hc-basic-suite-master-host0" target="_blank">http://jenkins.ovirt.org/job/o<wbr>virt_master_hc-system-tests/la<wbr>stCompletedBuild/artifact/expo<wbr>rted-artifacts/test_logs/hc-ba<wbr>sic-suite-master/post-002_boot<wbr>strap.py/lago-hc-basic-suite-m<wbr>aster-host0</a><br><br></pre><pre>thanks<span class="gmail-m_4346065651595802519gmail-m_3795450909556566802gmail-m_-511095684978464768HOEnZb"><font color="#888888"><br></font></span></pre><span class="gmail-m_4346065651595802519gmail-m_3795450909556566802gmail-m_-511095684978464768HOEnZb"><font color="#888888"><pre>sahina<br></pre></font></span></div>
<br></div></div>______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><span class="gmail-m_4346065651595802519gmail-m_3795450909556566802gmail-HOEnZb"><font color="#888888"><br></font></span></blockquote></div><span class="gmail-m_4346065651595802519gmail-m_3795450909556566802gmail-HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div class="gmail-m_4346065651595802519gmail-m_3795450909556566802gmail-m_-511095684978464768gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Sandro Bonazzola<br>Better technology. Faster innovation. Powered by community collaboration.<br>See how it works at <a href="http://redhat.com/" target="_blank">redhat.com</a></div></div></div></div></div></div></div></div>
</font></span></div>
</blockquote></div><br></div></div></div></div></div>
</blockquote></div><br></div></div>
______________________________<wbr>_________________<br>Devel mailing list<br><a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br><a href="http://lists.ovirt.org/mailman/listinfo/devel" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a></div></blockquote></div><br></div></div></div></blockquote></div><br></div></div>