[ovirt-users] ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Wed, Apr 26, 2017 at 1:28 PM, Simone Tiraboschi <stirabos at redhat.com>
wrote:

>
>
> On Wed, Apr 26, 2017 at 12:52 PM, Nir Soffer <nsoffer at redhat.com> wrote:
>
>> On Wed, Apr 26, 2017 at 11:36 AM Gianluca Cecchi <
>> gianluca.cecchi at gmail.com> wrote:
>>
>>> On Tue, Apr 25, 2017 at 11:28 PM, Nir Soffer <nsoffer at redhat.com> wrote:
>>>
>>>>
>>>>
>>>> Hi Gianluca,
>>>>
>>>> You can run this on the host:
>>>>
>>>> $ python -c "import yaml; print 'CLoader:', hasattr(yaml, 'CLoader')"
>>>> CLoader: True
>>>>
>>>> If you get "CLoader: False", you have some packaging issue, CLoader
>>>> is available on all supported platforms.
>>>>
>>>> Nir
>>>>
>>>>
>>>>> Thanks,
>>>>> Gianluca
>>>>>
>>>>>
>>>
>>> It seems ok.
>>>
>>> [root at ovirt01 ovirt-hosted-engine-ha]#  python -c "import yaml; print
>>> 'CLoader:', hasattr(yaml, 'CLoader')"
>>> CLoader: True
>>> [root at ovirt01 ovirt-hosted-engine-ha]#
>>>
>>> Anyway see here a sample of the spikes that it cntinues to have.. from
>>> 15% to 55% many times
>>> https://drive.google.com/file/d/0BwoPbcrMv8mvMy1xVUE3YzI2YVE
>>> /view?usp=sharing
>>>
>>
>> There are two issues in this video:
>> - Memory leak, ovirt-ha-agent is using 1g of memory. It is very unlikely
>> that it needs so much memory.
>> - Unusual cpu usage - but not the kind of usage related to yaml parsing.
>>
>> I would open two bugs for this. We have seen the first issue few month
>> ago, and
>> we did nothing about it so the memory leak was not fixed.
>>
>> To understand the unusual cpu usage, we need to integrate yappi into
>> ovirt-ha-agent,
>> and do some profiling to understand where cpu time is spent.
>>
>> Simone, can you do something based on these patches?
>> https://gerrit.ovirt.org/#/q/topic:generic-profiler
>>
>> I hope to get these patches merged soon.
>>
>>
> Absolutely at this point.
>
>

On 4.1.1, the 96% of the cpu time of ovirt-ha-agent is still spent in
connect() in /usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py
and the 95.98% is in Schema.__init__
in /usr/lib/python2.7/site-packages/vdsm/api/vdsmapi.py

So it's still the parsing of the api yaml schema.
On master we already merged a patch to reuse an existing connection if
available and this should mitigate/resolve the issue:
https://gerrit.ovirt.org/73757/

It's still not that clear why we are facing this kind of performance
regression.



> Nir
>>
>>
>>>
>>>
>>> The host is an Intel NUC6i5 with 32Gb of ram. There are the engine, an
>>> F25 guest and a C7 desktop VMs running, without doing almost anything.
>>>
>>> Gianluca
>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170426/8aed1c9c/attachment.html>