<div dir="ltr"><div><div><div><div><div><div><div>Martin,<br><br></div>Looking at the stats you generated I can see that there is almost no difference for cpu and memory. Load seems to be at the same level for both.<br></div>I tried to understand the differences by looking and # of calls and total time (top 10) but there was almost no difference. The only slight difference<br></div>I can see is in avg time. It seems that we haven't generate enough load to see significant improvement. How many client have you used during testing?<br><br></div>I am not sure how much it would take but what do you think about using doctor as caching service for the engine and run few tests. <br></div>I wonder what would be the result of such PoC.<br><br></div>Thanks,<br></div>Piotr<br><div><div><div><div><div><div><br></div></div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 1, 2015 at 3:25 PM, Martin Betak <span dir="ltr"><<a href="mailto:mbetak@redhat.com" target="_blank">mbetak@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi All,<br>
<br>
so I installed a few more plugins to the pgCluu and PostgreSQL itself.<br>
Now I have also the overall system load and total numbers for specific<br>
queries.<br>
<br>
If you look at the report, database 'engine' and 'Statement statistics'<br>
we can clearly see that the overwhelming majority of DB time is spent in<br>
GetVmsRunningOnVds() stored procedure.<br>
<br>
Turining on Doctor Rest with 5 second full-dump interval you can see<br>
that the calls used by DoctorCacheManager (GetAllFromVms, GetAllFromVds....)<br>
have hard time to add up to at least 1% of the overall load.<br>
<br>
Also you can see the 'System': CPU and memory statistics that those are largely<br>
unaffected by running Doctor service alongside engine.<br>
<br>
I also tried setting the full update interval to Doctor to 1 second to see how<br>
this would go - so essentially each second do a full dump of business entities -<br>
and this moved the overall Doctor overhead to ~2.5% of total DB load.<br>
<br>
Of course for the UI purposes interval in the range of 3-5 seconds should be more<br>
than acceptable in my opinion.<br>
<br>
As always - questions and remarks are more than welcome :-)<br>
<br>
Best regards,<br>
<br>
Martin<br>
<div><div class="h5"><br>
----- Original Message -----<br>
> From: "Martin Betak" <<a href="mailto:mbetak@redhat.com">mbetak@redhat.com</a>><br>
> To: "Piotr Kliczewski" <<a href="mailto:pkliczew@redhat.com">pkliczew@redhat.com</a>><br>
> Cc: "<a href="mailto:engine-devel@ovirt.org">engine-devel@ovirt.org</a>" <<a href="mailto:devel@ovirt.org">devel@ovirt.org</a>><br>
> Sent: Wednesday, September 30, 2015 4:52:12 PM<br>
> Subject: Re: [ovirt-devel] Doctor Rest PostgreSQL report<br>
><br>
> ----- Original Message -----<br>
> > From: "Piotr Kliczewski" <<a href="mailto:pkliczew@redhat.com">pkliczew@redhat.com</a>><br>
> > To: "Martin Betak" <<a href="mailto:mbetak@redhat.com">mbetak@redhat.com</a>><br>
> > Cc: "<a href="mailto:engine-devel@ovirt.org">engine-devel@ovirt.org</a>" <<a href="mailto:devel@ovirt.org">devel@ovirt.org</a>>, "Eli Mesika"<br>
> > <<a href="mailto:emesika@redhat.com">emesika@redhat.com</a>>, "Martin Perina"<br>
> > <<a href="mailto:mperina@redhat.com">mperina@redhat.com</a>><br>
> > Sent: Wednesday, September 30, 2015 3:29:28 PM<br>
> > Subject: Re: Doctor Rest PostgreSQL report<br>
> ><br>
> > Martin,<br>
> ><br>
> > For me it would be great to understand how cpu, memory changes over time<br>
> > for the engine. I would like to see the same for doctor service.<br>
> > I was not able to find it but it would be great to understand how many<br>
> > queries there were for both tests and how log it took to run them.<br>
><br>
> Yes, right now I'm looking for other tools to provide me exactly with that.<br>
> I just wanted to share the preliminary aggregated statistics.<br>
><br>
> ><br>
> > It would be good to understand implications of running doctor on the same<br>
> > machine as engine and on other machine.<br>
> ><br>
><br>
> Indeed, this is precisely what I'm testing. The attached reports were with<br>
> Doctor running on the same machine as the engine.<br>
><br>
> > Thanks,<br>
> > Piotr<br>
> ><br>
> ><br>
> > On Wed, Sep 30, 2015 at 3:13 PM, Martin Betak <<a href="mailto:mbetak@redhat.com">mbetak@redhat.com</a>> wrote:<br>
> ><br>
> > > Hi All,<br>
> > ><br>
> > > I performed a stress test using FakeVDSM environment with 200+ hosts<br>
> > > and 500+ VMs.<br>
> > ><br>
> > > Attached are generated HTML reports for this environment.<br>
> > > In both cases I tried to simulate some random load using existing<br>
> > > webadmin. In the '_doctor' case the simple connector from [1] was<br>
> > > running *in addition to* the legacy UI.<br>
> > ><br>
> > > The used pgCluu tool [2] which may be useful<br>
> > > for DB experts for some further insight.<br>
> > ><br>
> > > I wanted to send this out as soon as possible so we can better analyze<br>
> > > our current performance and the possible impact Doctor Rest<br>
> > > integration would have on the system.<br>
> > ><br>
> > > Please feel free to review the attached reports and/or suggest other<br>
> > > ways/tools how to better benchmark the DB load caused by Doctor Rest.<br>
> > ><br>
> > > Thank you very much.<br>
> > ><br>
> > > Best regards<br>
> > ><br>
> > > Martin<br>
> > ><br>
> > > [1] <a href="https://gerrit.ovirt.org/#/c/45233/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/45233/</a><br>
> > > [2] <a href="http://pgcluu.darold.net/" rel="noreferrer" target="_blank">http://pgcluu.darold.net/</a><br>
> ><br>
</div></div>> _______________________________________________<br>
> Devel mailing list<br>
> <a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/devel</a><br>
><br>
</blockquote></div><br></div>