[ovirt-users] Ovirt VM Performance abd CPU times

[Daniel Helgenberger]

On 29.10.2014 16:44, Xavier Naveira wrote:
> On 10/29/2014 04:29 PM, Xavier Naveira wrote:
>> On 10/29/2014 04:06 PM, Daniel Helgenberger wrote:
>>>
>>>
>>> On 29.10.2014 15:57, Xavier Naveira wrote:
>>>> On 10/29/2014 03:07 PM, Xavier Naveira wrote:
>>>>> On 10/29/2014 01:26 PM, Daniel Helgenberger wrote:
>>>>>> On 29.10.2014 11:48, Xavier Naveira wrote:
>>>>>>> On 10/29/2014 11:47 AM, Xavier Naveira wrote:
>>>>>>>> On 10/29/2014 11:40 AM, Daniel Helgenberger wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 29.10.2014 10:21, Xavier Naveira wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> We are migrating our ifrastructure from kvm+libvirt hypervisors to
>>>>>>>>>> ovirt.
>>>>>>>>>>
>>>>>>>>>> Everything is working fine but we're noticing that all the
>>>>>>>>>> qemu-kvm
>>>>>>>>>> processes in the hypervisors take a lot of CPU.
>>>>>>>>> Without further details of the workload this is hard tell. One
>>>>>>>>> Reason I
>>>>>>>>> can think of might be KSM [1]. Is it enabled on your cluster(s)?
>>>>>>>>> What is
>>>>>>>>> your mem over-commitment setting?
>>>>>>>>>
>>>>>>>>> Note, IIRC the KSM policy is currently hard coded; it will start at
>>>>>>>>> 80%
>>>>>>>>> host mem usage.
>>>>>>>>>
>>>>>>>>> [1] http://www.ovirt.org/Sla/host-mom-policy
>>>>>>>>>>
>>>>>>>>>> The typical example is an idle machine, running top from the
>>>>>>>>>> machine
>>>>>>>>>> itself it reports cpu use percentages below 10% and loads (with 2
>>>>>>>>>> processors) of 0.0x. The process running that machine in the
>>>>>>>>>> hypervisor
>>>>>>>>>> rports cpu uses in the order of the 80-100%.
>>>>>>>>>>
>>>>>>>>>> Should the values look like this? Why are the idle machines eating
>>>>>>>>>> up so
>>>>>>>>>> much CPU time?
>>>>>>>>>>
>>>>>>>>>> Thank you.
>>>>>>>>>> Xavier
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Users mailing list
>>>>>>>>>> Users at ovirt.org
>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>> Hi, thank you for the answer.
>>>>>>>
>>>>>>> I've been trying to work out some pattern and realized that the VMs
>>>>>>> using that much cpu all are Redhat 5.x, the Readhat 6.x doesn't
>>>>>>> exhibit
>>>>>>> this kind of high cpu use. (we run only redhat/centos 5.x/6.x on the
>>>>>>> cluster)
>>>>>> What OS are the hosts running? In case of EL6, make sure you have
>>>>>> tuned-0.2.19-13.el6.noarch installed [1].
>>>>>
>>>>> That's exactly the version we've in the hypervisors.
>>>>>
>>>>>>
>>>>>> To further investigate please post Engine, VDSM, libvirt and kernel
>>>>>> versions from the hosts.
>>>>>
>>>>> vdsm-xmlrpc-4.14.11.2-0.el6.noarch
>>>>> vdsm-cli-4.14.11.2-0.el6.noarch
>>>>> vdsm-python-4.14.11.2-0.el6.x86_64
>>>>> vdsm-4.14.11.2-0.el6.x86_64
>>>>> vdsm-python-zombiereaper-4.14.11.2-0.el6.noarch
>>>>>
>>>>> libvirt-client-0.10.2-29.el6_5.12.x86_64
>>>>> libvirt-0.10.2-29.el6_5.12.x86_64
>>>>> libvirt-lock-sanlock-0.10.2-29.el6_5.12.x86_64
>>>>> libvirt-python-0.10.2-29.el6_5.12.x86_64
>>>>>
>>>>> 2.6.32-431.23.3.el6.x86_64 #1 SMP Wed Jul 16 06:12:23 EDT 2014 x86_64
>>>>> x86_64 x86_64 GNU/Linux
>>>>>
>>>>>
>>>>>>
>>>>>> [1] https://access.redhat.com/solutions/358033
>>>>>>>
>>>>>>> I'll take a look to the KSM config.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Xavier
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>> Actually, this seems to be it. But I'm already at a newer kernel:
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=705082
>>> Well, I do not have such hardware so I never run into the issue. You
>>> could disable HT as I suspect your physical cores are less then 64?
>>>
>>> Your workload might differ but my VMs usually do not benefit from
>>> 'threaded' cores and I want HT disabled anyway. Also, you can check
>>> cluster settings and disable 'count threads as cores' if enabled. But I
>>> think this might not make any difference.
>>>>
>>>
>> Yeah, these are machines with 4 sockets, 6 core per socket and HT
>> enabled, so total 48 "CPU".
Good to know; yet the largest host I have has 32 (2 sockets, 8 cores, HT 
enabled) CPUs and is not showing this issue (at least I just looked and 
everything seems fine).
>>
>> So, are you implying that the problem is the number of "CPUs"? We were
>> hoping to add some more hypervisors to the cluster next week that have
>> even more cores...
>>
>> I can probably try to disable HT when we add the next hypervisor next
>> week but it feels that it'd be just a workaround?
Maybe, but not a bad one as you should not have any disadvantages.
>
> I opened a bug at redhat just in
> case:https://bugzilla.redhat.com/show_bug.cgi?id=1158547
I have to ask as I cannot see the BZ because I have no subscription any 
more. Against witch component did you open it?
>

-- 
Daniel Helgenberger
m box bewegtbild GmbH

P: +49/30/2408781-22
F: +49/30/2408781-10

ACKERSTR. 19
D-10115 BERLIN


www.m-box.de  www.monkeymen.tv

Geschäftsführer: Martin Retschitzegger / Michaela Göllner
Handeslregister: Amtsgericht Charlottenburg / HRB 112767