[ovirt-devel] [VDSM] scalable sampling benchmark

[Nir Soffer]

----- Original Message -----
> From: "Francesco Romani" <fromani at redhat.com>
> To: "Nir Soffer" <nsoffer at redhat.com>
> Cc: devel at ovirt.org, "Michal Skrivanek" <mskrivan at redhat.com>
> Sent: Friday, August 1, 2014 4:23:25 PM
> Subject: Re: [ovirt-devel] [VDSM] scalable sampling benchmark
> 
> ----- Original Message -----
> > From: "Nir Soffer" <nsoffer at redhat.com>
> > To: "Francesco Romani" <fromani at redhat.com>
> > Cc: devel at ovirt.org, "Michal Skrivanek" <mskrivan at redhat.com>
> > Sent: Friday, August 1, 2014 3:10:08 PM
> > Subject: Re: [ovirt-devel] [VDSM] scalable sampling benchmark
> 
> > > find attached graphs for CPU and memory profiling.
> > > Some stats on RHEL 6.5:
> > > 
> > > master cpu usage:
> > >                         samples% below
> > >             average%    10%     25%     50%     75%
> > > libvirt     74.101      0.083   0.083   3.172   52.922
> > > vdsm        44.211	3.506   33.556  70.618  84.641
> > > total       30.504	0.000   9.599   99.750  100.000
> > > 
> > > scalable_sampling cpu usage:
> > > 
> > >                         samples% below
> > >             average%    10%     25%     50%     75%
> > > libvirt     58.835	0.000   0.591   28.270  86.160
> > 
> > I wonder if we are using libvirt correctly - maybe our thread pool is too
> > small, keeping tasks in our queue, instead of letting libvirt process
> > them concurrently?
> > 
> > Can you check the size of the libvirt thread pool, and increase our
> > sampling
> > pool to the same value?
> 
> Will do.
> 
> > > vdsm        65.146	0.084   10.549  49.030  71.055
> > 
> > This looks 47% worse then the current code. We need a profile to understand
> > why.
> > 
> > Are you sure that we are doing the same amount of samplers in both cases?
> > Did you compare the logs?
> 
> I haven't yet did specific checks besides the code, but indeed this behaviour
> need a proper explanation, then I'll bump the priority of this task.
> Otherwise to run more benchs is close to useless.

It is not useless, maybe 200 vms case will give better results compared with
current code.