Re: [ovirt-devel] [VDSM] scalable sampling benchmark

From: "Francesco Romani" <fromani(a)redhat.com&gt; To: "Nir Soffer" <nsoffer(a)redhat.com&gt; Cc: devel(a)ovirt.org, "Michal Skrivanek" <mskrivan(a)redhat.com&gt; Sent: Friday, August 1, 2014 4:23:25 PM Subject: Re: [ovirt-devel] [VDSM] scalable sampling benchmark ----- Original Message ----- > From: "Nir Soffer" <nsoffer(a)redhat.com&gt; > To: "Francesco Romani" <fromani(a)redhat.com&gt; > Cc: devel(a)ovirt.org, "Michal Skrivanek" <mskrivan(a)redhat.com&gt; > Sent: Friday, August 1, 2014 3:10:08 PM > Subject: Re: [ovirt-devel] [VDSM] scalable sampling benchmark > > find attached graphs for CPU and memory profiling. > > Some stats on RHEL 6.5: > > > > master cpu usage: > > samples% below > > average% 10% 25% 50% 75% > > libvirt 74.101 0.083 0.083 3.172 52.922 > > vdsm 44.211 3.506 33.556 70.618 84.641 > > total 30.504 0.000 9.599 99.750 100.000 > > > > scalable_sampling cpu usage: > > > > samples% below > > average% 10% 25% 50% 75% > > libvirt 58.835 0.000 0.591 28.270 86.160 > > I wonder if we are using libvirt correctly - maybe our thread pool is too > small, keeping tasks in our queue, instead of letting libvirt process > them concurrently? > > Can you check the size of the libvirt thread pool, and increase our > sampling > pool to the same value? Will do. > > vdsm 65.146 0.084 10.549 49.030 71.055 > > This looks 47% worse then the current code. We need a profile to understand > why. > > Are you sure that we are doing the same amount of samplers in both cases? > Did you compare the logs? I haven't yet did specific checks besides the code, but indeed this behaviour need a proper explanation, then I'll bump the priority of this task. Otherwise to run more benchs is close to useless.