Re: [ovirt-devel] [VDSM] [JSONRPC] early, coarse grained benchmarks

From: "Nir Soffer" <nsoffer(a)redhat.com&gt; To: "Francesco Romani" <fromani(a)redhat.com&gt; Cc: &quot;engine-devel(a)ovirt.org&quot; <devel(a)ovirt.org&gt; Sent: Thursday, November 13, 2014 9:35:30 PM Subject: Re: [ovirt-devel] [VDSM] [JSONRPC] early, coarse grained benchmarks

> I'm profiling four major flows at the moment, and improving tools along the > way. > > Here's my series, still being worked on and not yet ready for review (still > draft for this reason) > > http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:... > > Here we have > 1. one patch to start/stop profiling at runtime using vdsClient Note that stopping yappi may segfault, as yappi is using unsafe iteration on Python internal link list without locking.

If we add such option, it should work only from local connection.

> http://gerrit.ovirt.org/#/c/35024/ > http://gerrit.ovirt.org/#/c/34989/ > http://gerrit.ovirt.org/#/c/35139 (DRAFT) > http://gerrit.ovirt.org/#/c/35140 (DRAFT) > > Let's see the effect of these in the #3 flows. > Scenario: > RHEL 6.5 on minidell running VDSM master + patches, with 150 VMs each one 1 What is the cpu usage and load during the test? In previous tests you had nice graph with this data.

This is not a good test - you just overload vdsm. Instead, run this verb in a more typical rate (like engine does) and limit the test by the running time, not by the number of calls. Running like this will show that the patched version needs less cpu and complete more calls during the test duration (e.g. 5 mintues).

> patched VDSM: What is "patched"? which patches are applied?

> http://gerrit.ovirt.org/#/c/35024/ > http://gerrit.ovirt.org/#/c/34989/ > http://gerrit.ovirt.org/#/c/35139 (DRAFT) > http://gerrit.ovirt.org/#/c/35140 (DRAFT)

This look strange: vanila vdsm: 75750 19.871 0.000 75.819 0.001 vm.py:2271(Vm._getRunningVmStats) patched vdsm: 234244 183.784 0.001 570.998 0.002 vm.py:2274(Vm._getRunningVmStats) The second profile is doing about 3 times more Vm._getRunningVmStats calls, which takes 10 times more cpu time(?!). We don't have any info on the load - maybe vdsm is overloaded in both runs, which can explain why it does 1/3 of the work vs the patched version, but the patched version is probably overloaded as well, which can explain the much higher cpu time. I suggest to check the cpu usage, and do *not* make profiles where vdsm uses more then 100% cpu. Lets profile much lower load to understand where time is taken.

> Open points: > * profile remaining flows again and report like this (in progress for > getAllVmStats) > * add wiki page > * find out why there is a so high minimum price for list verb > * profile JSONRPC (needs patches to vdsClient) Comparing jsonrpc would be very interesting - but we don't have a jsonrpc client yet, so you better use the engine for this intead of vdsclient.