[ovirt-devel] Doctor Rest PostgreSQL report

[Martin Betak]

Hi All,

so I installed a few more plugins to the pgCluu and PostgreSQL itself.
Now I have also the overall system load and total numbers for specific
queries.

If you look at the report, database 'engine' and 'Statement statistics'
we can clearly see that the overwhelming majority of DB time is spent in
GetVmsRunningOnVds() stored procedure.

Turining on Doctor Rest with 5 second full-dump interval you can see
that the calls used by DoctorCacheManager (GetAllFromVms, GetAllFromVds....)
have hard time to add up to at least 1% of the overall load.

Also you can see the 'System': CPU and memory statistics that those are largely
unaffected by running Doctor service alongside engine.

I also tried setting the full update interval to Doctor to 1 second to see how
this would go - so essentially each second do a full dump of business entities -
and this moved the overall Doctor overhead to ~2.5% of total DB load.

Of course for the UI purposes interval in the range of 3-5 seconds should be more
than acceptable in my opinion.

As always - questions and remarks are more than welcome :-)

Best regards,

Martin

----- Original Message -----
> From: "Martin Betak" <mbetak at redhat.com>
> To: "Piotr Kliczewski" <pkliczew at redhat.com>
> Cc: "engine-devel at ovirt.org" <devel at ovirt.org>
> Sent: Wednesday, September 30, 2015 4:52:12 PM
> Subject: Re: [ovirt-devel] Doctor Rest PostgreSQL report
> 
> ----- Original Message -----
> > From: "Piotr Kliczewski" <pkliczew at redhat.com>
> > To: "Martin Betak" <mbetak at redhat.com>
> > Cc: "engine-devel at ovirt.org" <devel at ovirt.org>, "Eli Mesika"
> > <emesika at redhat.com>, "Martin Perina"
> > <mperina at redhat.com>
> > Sent: Wednesday, September 30, 2015 3:29:28 PM
> > Subject: Re: Doctor Rest PostgreSQL report
> > 
> > Martin,
> > 
> > For me it would be great to understand how cpu, memory changes over time
> > for the engine. I would like to see the same for doctor service.
> > I was not able to find it but it would be great to understand how many
> > queries there were for both tests and how log it took to run them.
> 
> Yes, right now I'm looking for other tools to provide me exactly with that.
> I just wanted to share the preliminary aggregated statistics.
> 
> > 
> > It would be good to understand implications of running doctor on the same
> > machine as engine and on other machine.
> > 
> 
> Indeed, this is precisely what I'm testing. The attached reports were with
> Doctor running on the same machine as the engine.
> 
> > Thanks,
> > Piotr
> > 
> > 
> > On Wed, Sep 30, 2015 at 3:13 PM, Martin Betak <mbetak at redhat.com> wrote:
> > 
> > > Hi All,
> > >
> > > I performed a stress test using FakeVDSM environment with 200+ hosts
> > > and 500+ VMs.
> > >
> > > Attached are generated HTML reports for this environment.
> > > In both cases I tried to simulate some random load using existing
> > > webadmin. In the '_doctor' case the simple connector from [1] was
> > > running *in addition to* the legacy UI.
> > >
> > > The used pgCluu tool [2] which may be useful
> > > for DB experts for some further insight.
> > >
> > > I wanted to send this out as soon as possible so we can better analyze
> > > our current performance and the possible impact Doctor Rest
> > > integration would have on the system.
> > >
> > > Please feel free to review the attached reports and/or suggest other
> > > ways/tools how to better benchmark the DB load caused by Doctor Rest.
> > >
> > > Thank you very much.
> > >
> > > Best regards
> > >
> > > Martin
> > >
> > > [1] https://gerrit.ovirt.org/#/c/45233/
> > > [2] http://pgcluu.darold.net/
> > 
> _______________________________________________
> Devel mailing list
> Devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: report_nodoctor.tar.gz
Type: application/x-compressed-tar
Size: 593148 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20151001/a0546361/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: report_doctor.tar.gz
Type: application/x-compressed-tar
Size: 591329 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20151001/a0546361/attachment-0003.bin>