<div dir="ltr">Thanks Roman.<div>Adding Oved for Infra visibility. We have a lot to gain here.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 29, 2015 at 3:28 PM, Roman Mohr <span dir="ltr"><<a href="mailto:rmohr@redhat.com" target="_blank">rmohr@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Fri, Oct 2, 2015 at 4:24 PM, Michal Skrivanek <span dir="ltr"><<a href="mailto:michal.skrivanek@redhat.com" target="_blank">michal.skrivanek@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><br><div><span><div>On 2 Oct 2015, at 12:47, Roman Mohr wrote:</div><br><blockquote type="cite"><div dir="ltr">Hi All,<br><br>I am contributing to the engine for three months now. While I dug into the code I<br>started to wonder how to visualize what the engine is actually doing.<br></div></blockquote><div><br></div></span><div>This is one of the main problems with large application, anything to help to understand what's going on is very welcome</div><div><div><br><blockquote type="cite"><div dir="ltr"><br>To get better insights I added hystrix[1] to the engine. Hystrix is a circuit<br>breaker library which was developed by Netflix and has one pretty interesting<br>feature: Real time metrics for commands.<br><br>In combination with hystrix-dashboard[2] it allows very interesting insights.<br>You can easily get an overview of commands involved in operations, their<br>performance and complexity. Look at [2] and the attachments in [5] and [6] for<br>screenshots to get an Impression.<br><br>I want to propose to integrate hystrix permanently because from my perspective<br>the results were really useful and I also had some good experiences with hystrix<br>in past projects.<br><br>A first implementation can be found on gerrit[3].<br><br># Where is it immediately useful?<br><br>During development and QA.<br><br>An example: I tested the hystrix integration on /api/vms and /api/hosts rest<br>endpoints and immediately saw that the number of command exectuions grew<br>lineary whit the number of vms and hosts. The bug reports [5] and [6] are the<br>result.<br><br># How to monitor the engine?<br><br>It is as easy as starting a hystrix-dashboard [2] with<br><br> $ git clone <a href="https://github.com/Netflix/Hystrix.git" target="_blank">https://github.com/Netflix/Hystrix.git</a><br> $ cd Hystrix/hystrix-dashboard<br> $ ../gradlew jettyRun<br><br>and point the dashboard to <br><br> https://<customer.engine.ip>/ovirt-engine/hystrix.stream.<br><br># Other possible benefits?<br><br> * Live metrics at customer site for admins, consultants and support.<br><br> * Historical metrics for analysis in addition to the log files.<br> The metrics information is directly usable in graphite [7]. Therefore it would be<br> possible to collect the json stream for a certain time period and analyze them<br> later like in [4]. To do that someone just has to run<br><br> curl --user admin@internal:engine <a href="http://localhost:8080/ovirt-engine/api/hystrix.stream" target="_blank">http://localhost:8080/ovirt-engine/api/hystrix.stream</a> > hystrix.stream<br><br> for as long as necessary. The results can be analyzed later.<br></div></blockquote><div><br></div></div></div>+1</div><div>it's a great idea and when properly documented so even a BFU can do that it would allow us to get much better idea when something is not working or working too slow on a system we don't have access to, but it\'s reproducible elsewhere. Just ask for "hey, run this thingie while you are reproducing the issue and send us the result"</div><div><br><blockquote type="cite"><div dir="ltr"><span><br># Possible architectural benefits?<br><br>In addition to the live metrics we might also have use for the real hystrix features:<br><br> * Circuit Breaker<br> * Bulk execution of commands<br> * De-dublication of commands (Caching)<br> * Synchronous and asynchronous execution support<br> * ..<br><br></span><span>Our commands do already have a lot of features, so I don't think that there are<br>some quick wins, but maybe there are interesting opportunities for infra.<br></span></div></blockquote><div><br></div>eh..I would worry about that much later. First we should understand what are we actually doing and why (as we all know the engine is likely doing a lot of useless stuff;-)</div><div><span><br><blockquote type="cite"><div dir="ltr"><br># Overhead?<br><br>In [5] the netflix employees describe their results regarding the overhead of<br>wrapping every command into a new instance of a hystrix command.<br><br>They ran their tests on a standard 4-core Amazon EC2 server with a load of 60<br>request per second.<br><br>When using threadpools they measured a mean overhead of less than one<br>millisecond (so negligible). At the 90th percentile they measured an overhead<br>of 3 ms. At the 99th percentile of about 9 ms.<br></div></blockquote><div><br></div></span>This is likely good enough for backend commands and REST entry points (as you currently did), but may need more careful examination if we would want to add this to e.g. thread pool allocations</div><div>Don't get slowed down by that though, even for higher level stuff it is a great source of information</div><div><span><br><blockquote type="cite"><div dir="ltr"><br>When configuring the hystrix commands to use semaphores instead of threadpools<br>they are even faster.<br><br># How to integrate?<br><br>A working implementation can be found on gerrit[3]. These patch sets wrap a<br>hystrix command around every VdcAction, every VdcQuery and every VDSCommand.<br>This just required four small modifications in the code base.<br><br># Security?<br><br>In the provided patches the hystrix-metrics-servlet is accessible at<br>/ovirt-engine/api/hystrix.stream. It is protected by basic auth but accessible<br>for everyone who can authenticate. We should probably restrict it to admins.<br></div></blockquote><div><br></div></span>that would be great if it doesn't require too much work. If it does then we can start with enabling/disabling via JMX using Roy's recent patch [8]</div><div><span><br></span></div></div></blockquote><div><br></div></div></div><div>The hystrix stream is now accessible in http://<host>/ovirt-engine/services/hystrix.stream and admin privileges are needed.<br></div><div>Further it can be enabled an disabled via JMX (disabled by default). @Juan, @Roy thank you for your feedback on the code.<br> </div><span class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div><span><blockquote type="cite"><div dir="ltr"><br># Todo?<br><br>1) We do report failed actions with return values. Hystrix expects failing<br>commands to throw an exception. So on the dashboard almost every command looks<br>like a success. To overcome this, it would be pretty easy to throw an<br>exception inside the command and catch it immediately after it leaves the<br>hystrix wrapper.<br></div></blockquote><div><br></div></span>at the beginning it's probably enough to see what stuff is getting called, without differentiating between success or failure (we mostly do log failures, so hopefully we know when stuff is broken this way)</div><div><span><br></span></div></div></blockquote><div><br></div></span><div>Ok, I leave it disabled for now. But should really be just as easy as to throw an exception if the command fails and immediately catching it afterwards (not the nicest looking code then, but would work). And this can be encapsulated in the command executor, so it would not pollute the existing code.<br></div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div><span><blockquote type="cite"><div dir="ltr"><br>2) Finetuning<br>Do we want semaphores or a thread pool. When the thread pool, what size do we want? <br><br></div></blockquote></span></div></div></blockquote></span><div>To answer this myself, I use semaphores, to be sure to support transactions over multiple commands properly.<br></div><span class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div><span><blockquote type="cite"><div dir="ltr">3) Three unpackaged dependencies: archaius, hystrix-core, hystrix-contrib<br></div></blockquote><div><br></div></span><div>Since you yesterday volunteered to package them I think this should not stop us!:-)</div><div><br></div><div>thanks a lot for the effort, I miss a proper analysis for soooo long. Thanks for stepping up!</div><div><br></div>michal</div><div><span><br><blockquote type="cite"><div dir="ltr"><br># References<br><br>[1] <a href="https://github.com/Netflix/Hystrix" target="_blank">https://github.com/Netflix/Hystrix</a><br>[2] <a href="https://github.com/Netflix/Hystrix/tree/master/hystrix-dashboard" target="_blank">https://github.com/Netflix/Hystrix/tree/master/hystrix-dashboard</a> <br>[3] <a href="https://gerrit.ovirt.org/#/q/topic:hystrix" target="_blank">https://gerrit.ovirt.org/#/q/topic:hystrix</a> <br>[4] <a href="http://www.nurkiewicz.com/2015/02/storing-months-of-historical-metrics.html" target="_blank">http://www.nurkiewicz.com/2015/02/storing-months-of-historical-metrics.html</a><br>[5] <a href="https://github.com/Netflix/Hystrix/wiki/FAQ#what-is-the-processing-overhead-of-using-hystrix" target="_blank">https://github.com/Netflix/Hystrix/wiki/FAQ#what-is-the-processing-overhead-of-using-hystrix</a><br>[5] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1268216" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1268216</a> <br>[6] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1268224" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1268224</a><br>[7] <a href="http://graphite.wikidot.com" target="_blank">http://graphite.wikidot.com</a><br></div></blockquote></span>[8] <a href="https://gerrit.ovirt.org/#/c/29693/" target="_blank">https://gerrit.ovirt.org/#/c/29693/</a></div><div><br><blockquote type="cite">
_______________________________________________<br>Devel mailing list<br><a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br><a href="http://lists.ovirt.org/mailman/listinfo/devel" target="_blank">http://lists.ovirt.org/mailman/listinfo/devel</a></blockquote></div><br></div></blockquote></span></div><br></div></div>
<br>_______________________________________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/devel</a><br></blockquote></div><br></div>