Re: [ovirt-users] Ovirt VM Performance abd CPU times

On 10/29/2014 04:06 PM, Daniel Helgenberger wrote: > > > On 29.10.2014 15:57, Xavier Naveira wrote: >> On 10/29/2014 03:07 PM, Xavier Naveira wrote: >>> On 10/29/2014 01:26 PM, Daniel Helgenberger wrote: >>>> On 29.10.2014 11:48, Xavier Naveira wrote: >>>>> On 10/29/2014 11:47 AM, Xavier Naveira wrote: >>>>>> On 10/29/2014 11:40 AM, Daniel Helgenberger wrote: >>>>>>> >>>>>>> >>>>>>> On 29.10.2014 10:21, Xavier Naveira wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> We are migrating our ifrastructure from kvm+libvirt hypervisors to >>>>>>>> ovirt. >>>>>>>> >>>>>>>> Everything is working fine but we're noticing that all the >>>>>>>> qemu-kvm >>>>>>>> processes in the hypervisors take a lot of CPU. >>>>>>> Without further details of the workload this is hard tell. One >>>>>>> Reason I >>>>>>> can think of might be KSM [1]. Is it enabled on your cluster(s)? >>>>>>> What is >>>>>>> your mem over-commitment setting? >>>>>>> >>>>>>> Note, IIRC the KSM policy is currently hard coded; it will start at >>>>>>> 80% >>>>>>> host mem usage. >>>>>>> >>>>>>> [1] http://www.ovirt.org/Sla/host-mom-policy >>>>>>>> >>>>>>>> The typical example is an idle machine, running top from the >>>>>>>> machine >>>>>>>> itself it reports cpu use percentages below 10% and loads (with 2 >>>>>>>> processors) of 0.0x. The process running that machine in the >>>>>>>> hypervisor >>>>>>>> rports cpu uses in the order of the 80-100%. >>>>>>>> >>>>>>>> Should the values look like this? Why are the idle machines eating >>>>>>>> up so >>>>>>>> much CPU time? >>>>>>>> >>>>>>>> Thank you. >>>>>>>> Xavier >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list >>>>>>>> Users(a)ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>> >>>>>>> >>>>>> >>>>> Hi, thank you for the answer. >>>>> >>>>> I've been trying to work out some pattern and realized that the VMs >>>>> using that much cpu all are Redhat 5.x, the Readhat 6.x doesn't >>>>> exhibit >>>>> this kind of high cpu use. (we run only redhat/centos 5.x/6.x on the >>>>> cluster) >>>> What OS are the hosts running? In case of EL6, make sure you have >>>> tuned-0.2.19-13.el6.noarch installed [1]. >>> >>> That's exactly the version we've in the hypervisors. >>> >>>> >>>> To further investigate please post Engine, VDSM, libvirt and kernel >>>> versions from the hosts. >>> >>> vdsm-xmlrpc-4.14.11.2-0.el6.noarch >>> vdsm-cli-4.14.11.2-0.el6.noarch >>> vdsm-python-4.14.11.2-0.el6.x86_64 >>> vdsm-4.14.11.2-0.el6.x86_64 >>> vdsm-python-zombiereaper-4.14.11.2-0.el6.noarch >>> >>> libvirt-client-0.10.2-29.el6_5.12.x86_64 >>> libvirt-0.10.2-29.el6_5.12.x86_64 >>> libvirt-lock-sanlock-0.10.2-29.el6_5.12.x86_64 >>> libvirt-python-0.10.2-29.el6_5.12.x86_64 >>> >>> 2.6.32-431.23.3.el6.x86_64 #1 SMP Wed Jul 16 06:12:23 EDT 2014 x86_64 >>> x86_64 x86_64 GNU/Linux >>> >>> >>>> >>>> [1] https://access.redhat.com/solutions/358033 >>>>> >>>>> I'll take a look to the KSM config. >>>>> >>>>> Cheers, >>>>> >>>>> Xavier >>>>> >>>> >>> >> >> Actually, this seems to be it. But I'm already at a newer kernel: >> https://bugzilla.redhat.com/show_bug.cgi?id=705082 > Well, I do not have such hardware so I never run into the issue. You > could disable HT as I suspect your physical cores are less then 64? > > Your workload might differ but my VMs usually do not benefit from > 'threaded' cores and I want HT disabled anyway. Also, you can check > cluster settings and disable 'count threads as cores' if enabled. But I > think this might not make any difference. >> > Yeah, these are machines with 4 sockets, 6 core per socket and HT enabled, so total 48 "CPU". So, are you implying that the problem is the number of "CPUs"? We were hoping to add some more hypervisors to the cluster next week that have even more cores... I can probably try to disable HT when we add the next hypervisor next week but it feels that it'd be just a workaround?