Re: [ovirt-users] Ovirt VM Performance abd CPU times

On 10/29/2014 05:15 PM, Daniel Helgenberger wrote: > > > On 29.10.2014 16:44, Xavier Naveira wrote: > >> On 10/29/2014 04:29 PM, Xavier Naveira wrote: >> >>> On 10/29/2014 04:06 PM, Daniel Helgenberger wrote: >>> >>>> >>>> >>>> On 29.10.2014 15:57, Xavier Naveira wrote: >>>> >>>>> On 10/29/2014 03:07 PM, Xavier Naveira wrote: >>>>> >>>>>> On 10/29/2014 01:26 PM, Daniel Helgenberger wrote: >>>>>> >>>>>>> On 29.10.2014 11:48, Xavier Naveira wrote: >>>>>>> >>>>>>>> On 10/29/2014 11:47 AM, Xavier Naveira wrote: >>>>>>>> >>>>>>>>> On 10/29/2014 11:40 AM, Daniel Helgenberger wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 29.10.2014 10:21, Xavier Naveira wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> We are migrating our ifrastructure from kvm+libvirt hypervisors >>>>>>>>>>> to >>>>>>>>>>> ovirt. >>>>>>>>>>> >>>>>>>>>>> Everything is working fine but we're noticing that all the >>>>>>>>>>> qemu-kvm >>>>>>>>>>> processes in the hypervisors take a lot of CPU. >>>>>>>>>>> >>>>>>>>>> Without further details of the workload this is hard tell. One >>>>>>>>>> Reason I >>>>>>>>>> can think of might be KSM [1]. Is it enabled on your cluster(s)? >>>>>>>>>> What is >>>>>>>>>> your mem over-commitment setting? >>>>>>>>>> >>>>>>>>>> Note, IIRC the KSM policy is currently hard coded; it will start >>>>>>>>>> at >>>>>>>>>> 80% >>>>>>>>>> host mem usage. >>>>>>>>>> >>>>>>>>>> [1] http://www.ovirt.org/Sla/host-mom-policy >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The typical example is an idle machine, running top from the >>>>>>>>>>> machine >>>>>>>>>>> itself it reports cpu use percentages below 10% and loads (with >>>>>>>>>>> 2 >>>>>>>>>>> processors) of 0.0x. The process running that machine in the >>>>>>>>>>> hypervisor >>>>>>>>>>> rports cpu uses in the order of the 80-100%. >>>>>>>>>>> >>>>>>>>>>> Should the values look like this? Why are the idle machines >>>>>>>>>>> eating >>>>>>>>>>> up so >>>>>>>>>>> much CPU time? >>>>>>>>>>> >>>>>>>>>>> Thank you. >>>>>>>>>>> Xavier >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Users mailing list >>>>>>>>>>> Users(a)ovirt.org >>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> Hi, thank you for the answer. >>>>>>>> >>>>>>>> I've been trying to work out some pattern and realized that the VMs >>>>>>>> using that much cpu all are Redhat 5.x, the Readhat 6.x doesn't >>>>>>>> exhibit >>>>>>>> this kind of high cpu use. (we run only redhat/centos 5.x/6.x on >>>>>>>> the >>>>>>>> cluster) >>>>>>>> >>>>>>> What OS are the hosts running? In case of EL6, make sure you have >>>>>>> tuned-0.2.19-13.el6.noarch installed [1]. >>>>>>> >>>>>> >>>>>> That's exactly the version we've in the hypervisors. >>>>>> >>>>>> >>>>>>> To further investigate please post Engine, VDSM, libvirt and kernel >>>>>>> versions from the hosts. >>>>>>> >>>>>> >>>>>> vdsm-xmlrpc-4.14.11.2-0.el6.noarch >>>>>> vdsm-cli-4.14.11.2-0.el6.noarch >>>>>> vdsm-python-4.14.11.2-0.el6.x86_64 >>>>>> vdsm-4.14.11.2-0.el6.x86_64 >>>>>> vdsm-python-zombiereaper-4.14.11.2-0.el6.noarch >>>>>> >>>>>> libvirt-client-0.10.2-29.el6_5.12.x86_64 >>>>>> libvirt-0.10.2-29.el6_5.12.x86_64 >>>>>> libvirt-lock-sanlock-0.10.2-29.el6_5.12.x86_64 >>>>>> libvirt-python-0.10.2-29.el6_5.12.x86_64 >>>>>> >>>>>> 2.6.32-431.23.3.el6.x86_64 #1 SMP Wed Jul 16 06:12:23 EDT 2014 x86_64 >>>>>> x86_64 x86_64 GNU/Linux >>>>>> >>>>>> >>>>>> >>>>>>> [1] https://access.redhat.com/solutions/358033 >>>>>>> >>>>>>>> >>>>>>>> I'll take a look to the KSM config. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> >>>>>>>> Xavier >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> Actually, this seems to be it. But I'm already at a newer kernel: >>>>> https://bugzilla.redhat.com/show_bug.cgi?id=705082 >>>>> >>>> Well, I do not have such hardware so I never run into the issue. You >>>> could disable HT as I suspect your physical cores are less then 64? >>>> >>>> Your workload might differ but my VMs usually do not benefit from >>>> 'threaded' cores and I want HT disabled anyway. Also, you can check >>>> cluster settings and disable 'count threads as cores' if enabled. But I >>>> think this might not make any difference. >>>> >>>>> >>>>> >>>> Yeah, these are machines with 4 sockets, 6 core per socket and HT >>> enabled, so total 48 "CPU". >>> >> Good to know; yet the largest host I have has 32 (2 sockets, 8 cores, HT > enabled) CPUs and is not showing this issue (at least I just looked and > everything seems fine). > >> >>> So, are you implying that the problem is the number of "CPUs"? We were >>> hoping to add some more hypervisors to the cluster next week that have >>> even more cores... >>> >>> I can probably try to disable HT when we add the next hypervisor next >>> week but it feels that it'd be just a workaround? >>> >> Maybe, but not a bad one as you should not have any disadvantages. > >> >> I opened a bug at redhat just in >> case:https://bugzilla.redhat.com/show_bug.cgi?id=1158547 >> > I have to ask as I cannot see the BZ because I have no subscription any > more. Against witch component did you open it? > >> >> > I did as in the original bug: kernel, and then I took KVM as subsystem. _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users