<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 22 Feb 2017, at 13:53, Simone Tiraboschi <<a href="mailto:stirabos@redhat.com" class="">stirabos@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><br class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">On Wed, Feb 22, 2017 at 1:33 PM, Simone Tiraboschi <span dir="ltr" class=""><<a href="mailto:stirabos@redhat.com" target="_blank" class="">stirabos@redhat.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">When ovirt-ha-agent checks the status of the engine VM we get:<div class=""><pre style="" class="">2017-02-21 22:21:14,738-0500 ERROR (jsonrpc/2) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'2ccc0ef0-cc31-45b8-8e91-<wbr class="">a78fa4cad671'} (api:69)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-<wbr class="">packages/vdsm/common/api.py", line 67, in method
ret = func(*args, **kwargs)
File "/usr/share/vdsm/API.py", line 335, in getStats
vm = self.vm
File "/usr/share/vdsm/API.py", line 130, in vm
raise exception.NoSuchVM(vmId=self._<wbr class="">UUID)
NoSuchVM: Virtual machine does not exist: {'vmId': u'2ccc0ef0-cc31-45b8-8e91-<wbr class="">a78fa4cad671'}</pre><pre style="" class=""><br class=""></pre><pre class="">While in ovirt-ha-agent logs we have:<pre style="" class=""><pre class="">MainThread::INFO::2017-02-21 22:21:18,583::hosted_engine::<wbr class="">453::ovirt_hosted_engine_ha.<wbr class="">agent.hosted_engine.<wbr class="">HostedEngine::(start_<wbr class="">monitoring) Current state UnknownLocalVmState (score: 3400)</pre><pre class="">...</pre></pre><pre style="" class="">MainThread::INFO::2017-02-21 22:21:31,199::state_<wbr class="">decorators::25::ovirt_hosted_<wbr class="">engine_ha.agent.hosted_engine.<wbr class="">HostedEngine::(check) Unknown local engine vm status no actions taken</pre></pre>Probably it's a bug or a regression somewhere on master.</div></div></blockquote><div class=""><br class=""></div><div class="">On ovirt-ha-broker side the detection is based on a strict string match on the error message that is expected to be exactly 'Virtual machine does not exist' to set down status otherwise we set unknown status as in this case:</div><div class=""><a href="https://gerrit.ovirt.org/gitweb?p=ovirt-hosted-engine-ha.git;a=blob;f=ovirt_hosted_engine_ha/broker/submonitors/engine_health.py;h=d633cb860b811e84021221771bf706a9a4ac1d63;hb=refs/heads/master#l54" class="">https://gerrit.ovirt.org/gitweb?p=ovirt-hosted-engine-ha.git;a=blob;f=ovirt_hosted_engine_ha/broker/submonitors/engine_health.py;h=d633cb860b811e84021221771bf706a9a4ac1d63;hb=refs/heads/master#l54</a><br class=""></div><div class=""> </div><div class="">Adding Francesco here to understand if something has recently changed there on vdsm side.</div></div></div></div></div></blockquote><div><br class=""></div>That’s not a very robust code handling.</div><div>Yes, the text changed, the vm id was added.</div><div>And yes, it may change again any time I guess</div><div><br class=""></div><div><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class=""><div class="gmail-h5"><div class=""><br class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">On Wed, Feb 22, 2017 at 1:02 PM, Sandro Bonazzola <span dir="ltr" class=""><<a href="mailto:sbonazzo@redhat.com" target="_blank" class="">sbonazzo@redhat.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Adding Lev</div><div class="gmail_extra"><br class=""><div class="gmail_quote"><div class=""><div class="gmail-m_3795450909556566802gmail-h5">On Wed, Feb 22, 2017 at 12:59 PM, Sahina Bose <span dir="ltr" class=""><<a href="mailto:sabose@redhat.com" target="_blank" class="">sabose@redhat.com</a>></span> wrote:<br class=""></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class="gmail-m_3795450909556566802gmail-h5"><div dir="ltr" class=""><pre class="">Hi all,<br class=""><br class="">On the HC setup, the HE VM is not restarted.<br class="">The agent.log has <br class="">MainThread::INFO::2017-02-21 22:09:58,022::state_machine::1<wbr class="">69::ovirt_hosted_engine_ha.age<wbr class="">nt.hosted_engine.HostedEngine:<wbr class="">:(refresh) Global metadata: {}
MainThread::INFO::2017-02-21 22:09:58,023::state_machine::1<wbr class="">77::ovirt_hosted_engine_ha.age<wbr class="">nt.hosted_engine.HostedEngine:<wbr class="">:(refresh) Local (id 1): {'engine-health': {'reason': 'failed to getVmStats', 'health': 'unknown', 'vm': 'unknown', 'detail': 'unknown'}, 'bridge': True, 'mem-free': 4079.0, 'maintenance': False, 'cpu-load': 0.0491, 'gateway': True}<br class="">...<br class="">MainThread::INFO::2017-02-21 22:10:29,219::state_decorators<wbr class="">::25::ovirt_hosted_engine_ha.a<wbr class="">gent.hosted_engine.HostedEngin<wbr class="">e::(check) Unknown local engine vm status no actions taken
MainThread::INFO::2017-02-21 22:10:29,219::brokerlink::111:<wbr class="">:<a href="http://ovirt_hosted_engine_ha.lib.br/" target="_blank" class="">ovirt_hosted_engine_ha.lib.br</a><wbr class="">okerlink.BrokerLink::(notify) Trying: notify time=1487733029.22 type=state_transition detail=ReinitializeFSM-Unknown<wbr class="">LocalVmState hostname='lago-hc-basic-suite-<wbr class="">master-host0'
MainThread::INFO::2017-02-21 22:10:29,317::brokerlink::121:<wbr class="">:<a href="http://ovirt_hosted_engine_ha.lib.br/" target="_blank" class="">ovirt_hosted_engine_ha.lib.br</a><wbr class="">okerlink.BrokerLink::(notify) Success, was notification of state_transition (ReinitializeFSM-UnknownLocalV<wbr class="">mState) sent? ignored<br class=""></pre><pre class="">and the vdsm.log <br class=""><br class="">2017-02-21 22:09:11,962-0500 INFO (libvirt/events) [virt.vm] (vmId='2ccc0ef0-cc31-45b8-8e91<wbr class="">-a78fa4cad671') Changed state to Down: User shut down from within the guest (code=7) (vm:1269)
2017-02-21 22:09:11,962-0500 INFO (libvirt/events) [virt.vm] (vmId='2ccc0ef0-cc31-45b8-8e91<wbr class="">-a78fa4cad671') Stopping connection (guestagent:429)<br class=""><br class="">2017-02-21 22:09:29,727-0500 ERROR (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'2ccc0ef0-cc31-45b8-8e91-a78f<wbr class="">a4cad671'} (api:69)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packa<wbr class="">ges/vdsm/common/api.py", line 67, in method
ret = func(*args, **kwargs)
File "/usr/share/vdsm/API.py", line 335, in getStats
vm = self.vm
File "/usr/share/vdsm/API.py", line 130, in vm
raise exception.NoSuchVM(vmId=self._<wbr class="">UUID)
NoSuchVM: Virtual machine does not exist: {'vmId': u'2ccc0ef0-cc31-45b8-8e91-a78f<wbr class="">a4cad671'}<br class=""><br class=""><br class=""></pre><pre class="">What should I be looking for to identify the issue?<br class=""><br class=""></pre><pre class="">The logs are at <a href="http://jenkins.ovirt.org/job/ovirt_master_hc-system-tests/lastCompletedBuild/artifact/exported-artifacts/test_logs/hc-basic-suite-master/post-002_bootstrap.py/lago-hc-basic-suite-master-host0" target="_blank" class="">http://jenkins.ovirt.org/job/o<wbr class="">virt_master_hc-system-tests/la<wbr class="">stCompletedBuild/artifact/expo<wbr class="">rted-artifacts/test_logs/hc-ba<wbr class="">sic-suite-master/post-002_boot<wbr class="">strap.py/lago-hc-basic-suite-<wbr class="">master-host0</a><br class=""><br class=""></pre><pre class="">thanks<span class="gmail-m_3795450909556566802gmail-m_-511095684978464768HOEnZb"><font color="#888888" class=""><br class=""></font></span></pre><span class="gmail-m_3795450909556566802gmail-m_-511095684978464768HOEnZb"><font color="#888888" class=""><pre class="">sahina<br class=""></pre></font></span></div>
<br class=""></div></div>______________________________<wbr class="">_________________<br class="">
Devel mailing list<br class="">
<a href="mailto:Devel@ovirt.org" target="_blank" class="">Devel@ovirt.org</a><br class="">
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank" class="">http://lists.ovirt.org/mailman<wbr class="">/listinfo/devel</a><span class="gmail-m_3795450909556566802gmail-HOEnZb"><font color="#888888" class=""><br class=""></font></span></blockquote></div><span class="gmail-m_3795450909556566802gmail-HOEnZb"><font color="#888888" class=""><br class=""><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div class="gmail-m_3795450909556566802gmail-m_-511095684978464768gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class="">Sandro Bonazzola<br class="">Better technology. Faster innovation. Powered by community collaboration.<br class="">See how it works at <a href="http://redhat.com/" target="_blank" class="">redhat.com</a></div></div></div></div></div></div></div></div>
</font></span></div>
</blockquote></div><br class=""></div></div></div></div></div>
</blockquote></div><br class=""></div></div>
_______________________________________________<br class="">Devel mailing list<br class=""><a href="mailto:Devel@ovirt.org" class="">Devel@ovirt.org</a><br class="">http://lists.ovirt.org/mailman/listinfo/devel</div></blockquote></div><br class=""></body></html>