Re: [ovirt-users] Ovirt VM Performance abd CPU times

On our clusters with pure computation we use always HT off, HT was slowing down scientific calculations. Just I'm interested if any benefit of enabling HT on the Ovirt host? At least on my system host-Centos6.5, guest-Centos7VM-LAMPstack with separate CentOS7-MariaDBVM did not show any performance boost if I switch on/off HT on 2650v2. a. On Wed, Oct 29, 2014 at 5:24 PM, Xavier Naveira <xnaveira(a)gmail.com&gt; wrote: > On 10/29/2014 05:15 PM, Daniel Helgenberger wrote: > >> >> >> On 29.10.2014 16:44, Xavier Naveira wrote: >> >>> On 10/29/2014 04:29 PM, Xavier Naveira wrote: >>> >>>> On 10/29/2014 04:06 PM, Daniel Helgenberger wrote: >>>> >>>>> >>>>> >>>>> On 29.10.2014 15:57, Xavier Naveira wrote: >>>>> >>>>>> On 10/29/2014 03:07 PM, Xavier Naveira wrote: >>>>>> >>>>>>> On 10/29/2014 01:26 PM, Daniel Helgenberger wrote: >>>>>>> >>>>>>>> On 29.10.2014 11:48, Xavier Naveira wrote: >>>>>>>> >>>>>>>>> On 10/29/2014 11:47 AM, Xavier Naveira wrote: >>>>>>>>> >>>>>>>>>> On 10/29/2014 11:40 AM, Daniel Helgenberger wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 29.10.2014 10:21, Xavier Naveira wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> We are migrating our ifrastructure from kvm+libvirt >>>>>>>>>>>> hypervisors to >>>>>>>>>>>> ovirt. >>>>>>>>>>>> >>>>>>>>>>>> Everything is working fine but we're noticing that all the >>>>>>>>>>>> qemu-kvm >>>>>>>>>>>> processes in the hypervisors take a lot of CPU. >>>>>>>>>>>> >>>>>>>>>>> Without further details of the workload this is hard tell. One >>>>>>>>>>> Reason I >>>>>>>>>>> can think of might be KSM [1]. Is it enabled on your cluster(s)? >>>>>>>>>>> What is >>>>>>>>>>> your mem over-commitment setting? >>>>>>>>>>> >>>>>>>>>>> Note, IIRC the KSM policy is currently hard coded; it will >>>>>>>>>>> start at >>>>>>>>>>> 80% >>>>>>>>>>> host mem usage. >>>>>>>>>>> >>>>>>>>>>> [1] http://www.ovirt.org/Sla/host-mom-policy >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> The typical example is an idle machine, running top from the >>>>>>>>>>>> machine >>>>>>>>>>>> itself it reports cpu use percentages below 10% and loads >>>>>>>>>>>> (with 2 >>>>>>>>>>>> processors) of 0.0x. The process running that machine in the >>>>>>>>>>>> hypervisor >>>>>>>>>>>> rports cpu uses in the order of the 80-100%. >>>>>>>>>>>> >>>>>>>>>>>> Should the values look like this? Why are the idle machines >>>>>>>>>>>> eating >>>>>>>>>>>> up so >>>>>>>>>>>> much CPU time? >>>>>>>>>>>> >>>>>>>>>>>> Thank you. >>>>>>>>>>>> Xavier >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Users mailing list >>>>>>>>>>>> Users(a)ovirt.org >>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> Hi, thank you for the answer. >>>>>>>>> >>>>>>>>> I've been trying to work out some pattern and realized that the >>>>>>>>> VMs >>>>>>>>> using that much cpu all are Redhat 5.x, the Readhat 6.x doesn't >>>>>>>>> exhibit >>>>>>>>> this kind of high cpu use. (we run only redhat/centos 5.x/6.x on >>>>>>>>> the >>>>>>>>> cluster) >>>>>>>>> >>>>>>>> What OS are the hosts running? In case of EL6, make sure you have >>>>>>>> tuned-0.2.19-13.el6.noarch installed [1]. >>>>>>>> >>>>>>> >>>>>>> That's exactly the version we've in the hypervisors. >>>>>>> >>>>>>> >>>>>>>> To further investigate please post Engine, VDSM, libvirt and kernel >>>>>>>> versions from the hosts. >>>>>>>> >>>>>>> >>>>>>> vdsm-xmlrpc-4.14.11.2-0.el6.noarch >>>>>>> vdsm-cli-4.14.11.2-0.el6.noarch >>>>>>> vdsm-python-4.14.11.2-0.el6.x86_64 >>>>>>> vdsm-4.14.11.2-0.el6.x86_64 >>>>>>> vdsm-python-zombiereaper-4.14.11.2-0.el6.noarch >>>>>>> >>>>>>> libvirt-client-0.10.2-29.el6_5.12.x86_64 >>>>>>> libvirt-0.10.2-29.el6_5.12.x86_64 >>>>>>> libvirt-lock-sanlock-0.10.2-29.el6_5.12.x86_64 >>>>>>> libvirt-python-0.10.2-29.el6_5.12.x86_64 >>>>>>> >>>>>>> 2.6.32-431.23.3.el6.x86_64 #1 SMP Wed Jul 16 06:12:23 EDT 2014 >>>>>>> x86_64 >>>>>>> x86_64 x86_64 GNU/Linux >>>>>>> >>>>>>> >>>>>>> >>>>>>>> [1] https://access.redhat.com/solutions/358033 >>>>>>>> >>>>>>>>> >>>>>>>>> I'll take a look to the KSM config. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> >>>>>>>>> Xavier >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> Actually, this seems to be it. But I'm already at a newer kernel: >>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=705082 >>>>>> >>>>> Well, I do not have such hardware so I never run into the issue. You >>>>> could disable HT as I suspect your physical cores are less then 64? >>>>> >>>>> Your workload might differ but my VMs usually do not benefit from >>>>> 'threaded' cores and I want HT disabled anyway. Also, you can check >>>>> cluster settings and disable 'count threads as cores' if enabled. But >>>>> I >>>>> think this might not make any difference. >>>>> >>>>>> >>>>>> >>>>> Yeah, these are machines with 4 sockets, 6 core per socket and HT >>>> enabled, so total 48 "CPU". >>>> >>> Good to know; yet the largest host I have has 32 (2 sockets, 8 cores, HT >> enabled) CPUs and is not showing this issue (at least I just looked and >> everything seems fine). >> >>> >>>> So, are you implying that the problem is the number of "CPUs"? We were >>>> hoping to add some more hypervisors to the cluster next week that have >>>> even more cores... >>>> >>>> I can probably try to disable HT when we add the next hypervisor next >>>> week but it feels that it'd be just a workaround? >>>> >>> Maybe, but not a bad one as you should not have any disadvantages. >> >>> >>> I opened a bug at redhat just in >>> case:https://bugzilla.redhat.com/show_bug.cgi?id=1158547 >>> >> I have to ask as I cannot see the BZ because I have no subscription any >> more. Against witch component did you open it? >> >>> >>> >> I did as in the original bug: kernel, and then I took KVM as subsystem. > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >