<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 26, 2017 at 1:28 PM, Simone Tiraboschi <span dir="ltr"><<a href="mailto:stirabos@redhat.com" target="_blank">stirabos@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="gmail-h5">On Wed, Apr 26, 2017 at 12:52 PM, Nir Soffer <span dir="ltr"><<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span><div class="gmail_quote"><div dir="ltr">On Wed, Apr 26, 2017 at 11:36 AM Gianluca Cecchi <<a href="mailto:gianluca.cecchi@gmail.com" target="_blank">gianluca.cecchi@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Apr 25, 2017 at 11:28 PM, Nir Soffer <span dir="ltr"><<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_quote"><div><div class="gmail-m_-5101781545891541948m_-3234011586418188284m_1409115595870497153m_-1514610739421722293gmail-h5"><div><br></div></div></div><div>Hi Gianluca,</div><div><br></div><div>You can run this on the host:</div><div><br></div><div>$ python -c "import yaml; print 'CLoader:', hasattr(yaml, 'CLoader')"</div><div>CLoader: True</div><div><br></div><div>If you get "CLoader: False", you have some packaging issue, CLoader</div><div>is available on all supported platforms.</div><div><br></div><div>Nir</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"></div><div class="gmail_extra">Thanks,<br></div><div class="gmail_extra">Gianluca<br></div><div class="gmail_extra"><br></div></div>
</blockquote></div></div>
</blockquote></div><br></div><div class="gmail_extra"><br></div></div><div dir="ltr"><div class="gmail_extra">It seems ok.</div><div class="gmail_extra"><br></div><div class="gmail_extra"><div class="gmail_extra">[root@ovirt01 ovirt-hosted-engine-ha]# python -c "import yaml; print 'CLoader:', hasattr(yaml, 'CLoader')"</div><div class="gmail_extra">CLoader: True</div><div class="gmail_extra">[root@ovirt01 ovirt-hosted-engine-ha]# </div><div><br></div><div>Anyway see here a sample of the spikes that it cntinues to have.. from 15% to 55% many times</div><div><a href="https://drive.google.com/file/d/0BwoPbcrMv8mvMy1xVUE3YzI2YVE/view?usp=sharing" target="_blank">https://drive.google.com/file/<wbr>d/0BwoPbcrMv8mvMy1xVUE3YzI2YVE<wbr>/view?usp=sharing</a></div></div></div></blockquote><div><br></div></div></span><div dir="ltr"><div class="gmail_quote"><div>There are two issues in this video:</div><div>- Memory leak, ovirt-ha-agent is using 1g of memory. It is very unlikely that it needs so much memory.</div><div>- Unusual cpu usage - but not the kind of usage related to yaml parsing.</div><div><br></div><div>I would open two bugs for this. We have seen the first issue few month ago, and</div><div>we did nothing about it so the memory leak was not fixed.</div><div><br></div><div>To understand the unusual cpu usage, we need to integrate yappi into ovirt-ha-agent,</div><div>and do some profiling to understand where cpu time is spent.</div><div><br></div><div>Simone, can you do something based on these patches?</div><div><a href="https://gerrit.ovirt.org/#/q/topic:generic-profiler" target="_blank">https://gerrit.ovirt.org/#/q/t<wbr>opic:generic-profiler</a><br></div><div><br></div><div>I hope to get these patches merged soon.</div></div></div><span class="gmail-m_-5101781545891541948HOEnZb"><font color="#888888"><div dir="ltr"><div class="gmail_quote"><div><br></div></div></div></font></span></div></blockquote><div><br></div></div></div><div>Absolutely at this point.</div><span class="gmail-"><div> </div></span></div></div></div></blockquote><div><br></div><div>On 4.1.1, the 96% of the cpu time of ovirt-ha-agent is still spent in connect() in /usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py</div><div>and the 95.98% is in Schema.__init__ in /usr/lib/python2.7/site-packages/vdsm/api/vdsmapi.py</div><div><br></div><div>So it's still the parsing of the api yaml schema.</div><div>On master we already merged a patch to reuse an existing connection if available and this should mitigate/resolve the issue: <a href="https://gerrit.ovirt.org/73757/">https://gerrit.ovirt.org/73757/</a></div><div><br></div><div>It's still not that clear why we are facing this kind of performance regression.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="gmail-"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span class="gmail-m_-5101781545891541948HOEnZb"><font color="#888888"><div dir="ltr"><div class="gmail_quote"><div></div><div>Nir </div></div></div></font></span><span><div dir="ltr"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div><br></div><div><br></div><div>The host is an Intel NUC6i5 with 32Gb of ram. There are the engine, an F25 guest and a C7 desktop VMs running, without doing almost anything.</div></div></div><div dir="ltr"><div class="gmail_extra"><div><br></div><div>Gianluca</div><div><br></div></div></div></blockquote></div></div></span></div>
</blockquote></span></div><br></div></div>
</blockquote></div><br></div></div>