<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 27, 2017 at 1:32 PM, Juan Hernández <span dir="ltr"><<a href="mailto:jhernand@redhat.com" target="_blank">jhernand@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-">On 03/27/2017 01:03 PM, Tomas Jelinek wrote:<br>
><br>
><br>
> On Mon, Mar 27, 2017 at 11:21 AM, Juan Hernández <<a href="mailto:jhernand@redhat.com">jhernand@redhat.com</a><br>
</span><div><div class="gmail-h5">> <mailto:<a href="mailto:jhernand@redhat.com">jhernand@redhat.com</a>>> wrote:<br>
><br>
> Top posting, sorry.<br>
><br>
> There are a few things I'd like to clarify, regarding this subject:<br>
><br>
> 1. Data aggregation, as requested now by Tomas, and by other people in<br>
> the past.<br>
><br>
> We used to have that 'detail' parameter, to aggregate certain very<br>
> specific types of data, in particular to aggregate VM disks and NICs. We<br>
> removed that in version 4 of the API because the implementation was<br>
> extremely inefficient, from the engine point of view. An innocent<br>
> request like this:<br>
><br>
> GET /ovirt-engine/api/vms?detail=+<wbr>disks,+nics<br>
><br>
> Would generate, with the implementation we used to have, 1 query for the<br>
> VMs and then as many queries for disks and NICs as VMs in the system. In<br>
> our scale test environments, for example, with approx 4000 VMs and 10000<br>
> disks, that would take more than 20 hours to execute.<br>
><br>
> In addition, we didn't have in the past any mechanism to make this<br>
> available in a generic one, because there was no knowledge in the API of<br>
> what are 'details'.<br>
><br>
> In version 4 of the API we introduced a formal (kind of) specification<br>
> of the API (a.k.a. the model), and int includes knowledge about what are<br>
> 'links'. For example, the specification of the VM type contains this:<br>
><br>
> @Link DiskAttachment[] diskAttachments();<br>
> @Link Nic[] nics();<br>
><br>
> With this information we are now in a position where we can implement<br>
> this in a generic way.<br>
><br>
> We intend to implement this using a mechanism similar to the existing<br>
> 'detail' parameter:<br>
><br>
> GET /ovirt-engine/api/vms/123?<wbr>follow=disk_attachments,nics<br>
><br>
> The naive implementation of this is to let the API call itself. For<br>
> example, when the user requests to follow the 'disk_attachments' detail<br>
> the API can just call itself to get that:<br>
><br>
> GET /ovirt-engine/api/vms/123/<wbr>disk_attachments<br>
><br>
> However, we can't use that naive approach, if we do we end with the<br>
> 1+C*N query problem described before. We need to use specific<br>
> implementations for certain frequent use cases, like VMs+disks+nics, and<br>
> that needs work in the API and in the backend.<br>
><br>
> Tomas, if you want to help moving this forward, please open a RFE and<br>
> makes sure it gets attention.<br></div></div></blockquote><div><br></div><div>ok, opened: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1436206">https://bugzilla.redhat.com/show_bug.cgi?id=1436206</a></div><div>Will try to get it done soon.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-h5">
><br>
><br>
> This sounds pretty good! I will open, but since we are talking already<br>
> here I'll just use the opportunity to clarify the topic more and than<br>
> I'll open the BZ.<br>
><br>
> What I can imagine is the GetAllVmsQuery will accept in params also the<br>
> list of details it should provide. Than, the GetAllVmsQuery will<br>
> implement the efficient way of retrieving this info as well.<br>
><br>
> So, from the API perspective, it will be about taking the<br>
> ?follow=<something> part and passing it to the backend query params.<br>
><br>
> What you think?<br>
><br>
<br>
</div></div>Exactly, that is the point! The API by itself can't optimize database<br>
queries, all it can do is call the backend. It is the backend that has<br>
the opportunity and possibility to send optimized queries to the database.<br>
<br>
For other less common things we can use the naive approach, and<br>
implement the aggregation in the API itself. But for common use cases,<br>
like VM+disks+nics, we need to do it in an efficient way.<br>
<div><div class="gmail-h5"><br>
><br>
><br>
> 2. Reuse of TLS sessions.<br>
><br>
> The part of creating TLS sessions that is expensive is the generation of<br>
> the shared session key. That can be avoided if both the server and the<br>
> client are careful and reuse the session, using the session cache<br>
> mechanism built-in into TLS itself. The web servers that we use (Apache<br>
> and Undertow) do implement this mechanism, and so do most of our<br>
> clients. Make sure that your client uses it as well. In Java this is<br>
> achieved re-using the SSLContext. We already do that for the engine to<br>
> VDSM communciation for example. In JavaScript the browser already takes<br>
> care of this.<br>
><br>
> 3. Parallelism and latency.<br>
><br>
> A typical problem that we have is that we send many request to the<br>
> server. For example, to retrieve user sessions for a set of VMs we tend<br>
> to send many requests like this:<br>
><br>
> GET /ovirt-engine/api/vms/1/<wbr>sessions<br>
> GET /ovirt/engine/api/vms/2/<wbr>sessions<br>
> GET /ovirt-engine/api/vms/3/<wbr>sessions<br>
> ...<br>
><br>
> And we do that in a synchronous way: send one, wait for the result, send<br>
> another one, wait for the result, etc. This means that we don't take<br>
> advantage of the parallelism of the server and that we add to each<br>
> request the network round trip time. So if we have N requests, we have<br>
> to wait at least N*RTT.<br>
><br>
> The web servers that we use support multiple connections, and the<br>
> protocol that we use, HTTP, supports pipe-lining. This means that you<br>
> can send multiple requests in parallel, and that you can send multiple<br>
> requests without waiting for the response. To give you an idea of the<br>
> improvement that can be achieved, we recently added asynchronous request<br>
> support to the Ruby SDK, with multiple connections and pipe-lining. In<br>
> our scale testing environment that reduced the time to collect a<br>
> complete inventory from approx 30 min to approx 2 min. Here you have an<br>
> example:<br>
><br>
><br>
> <a href="https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/sdk/examples/asynchronous_inventory.rb" rel="noreferrer" target="_blank">https://github.com/oVirt/<wbr>ovirt-engine-sdk-ruby/blob/<wbr>master/sdk/examples/<wbr>asynchronous_inventory.rb</a><br>
> <<a href="https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/sdk/examples/asynchronous_inventory.rb" rel="noreferrer" target="_blank">https://github.com/oVirt/<wbr>ovirt-engine-sdk-ruby/blob/<wbr>master/sdk/examples/<wbr>asynchronous_inventory.rb</a>><br>
><br>
> So make sure that you take advantage of that in your clients. Sadly<br>
> pipe-lining is disabled by default in most browsers, so this isn't<br>
> helpful for JavaScript applications.<br>
><br>
><br>
> But we can try what we can do in moVirt about<br>
> this: <a href="https://github.com/oVirt/moVirt/issues/260" rel="noreferrer" target="_blank">https://github.com/oVirt/<wbr>moVirt/issues/260</a><br>
><br>
<br>
</div></div>Sure, I think there are plenty of asynchronous HTTP clients for Android,<br>
worth trying one of them. If you are brave enough you can even consider<br>
using the same library used in the Ruby SDK: libcurl. A bit of JNI here<br>
and there, and you are done.<br>
<span class="gmail-"><br>
><br>
><br>
> 4. HTTP/2 support.<br>
><br>
> The application server that we use, WildFly, supports HTTP/2, including<br>
> ALPN, out of the box, since version 10.1. We need a mechanism to<br>
> enable it:<br>
><br>
> core: Add support for enabling HTTP/2<br>
> <a href="https://gerrit.ovirt.org/74621" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/74621</a><br>
><br>
> And then we need to get Apache out of the way, for API traffic, at<br>
> least. I think that is something we can do in the context of the engine<br>
> "podification" effort.<br>
><br>
> However, note that HTTP/2 won't have that big impact in performance for<br>
> applications that continue to use a synchronous/serial style of<br>
> interaction with the API.<br>
><br>
> On 03/24/2017 11:16 PM, Yaniv Kaul wrote:<br>
> ><br>
> ><br>
> > On Fri, Mar 24, 2017 at 8:57 PM, Martin Sivak <<a href="mailto:msivak@redhat.com">msivak@redhat.com</a> <mailto:<a href="mailto:msivak@redhat.com">msivak@redhat.com</a>><br>
</span><span class="gmail-">> > <mailto:<a href="mailto:msivak@redhat.com">msivak@redhat.com</a> <mailto:<a href="mailto:msivak@redhat.com">msivak@redhat.com</a>>>> wrote:<br>
> ><br>
> > > Current Apache used has only experimental module for it.<br>
> > > Undertow is supposed to have a better support. I wonder when/if we can drop<br>
> > > Apache...<br>
> ><br>
> > The last info I have about that from mperina is that we need Apache<br>
> > for kerberos support atm.<br>
> ><br>
> ><br>
> > I don't think we need it - I remember reading that Undertow does support<br>
> > it as well.<br>
> > The only issue is that there are probably 10 people in the world who<br>
> > know how to configure Undertow for Kerberos, while many do for Apache.<br>
> > And since we leave it for the user to configure...<br>
> > Y.<br>
> ><br>
> ><br>
> ><br>
> > Martin<br>
> ><br>
> > On Fri, Mar 24, 2017 at 5:30 PM, Yaniv Kaul <<a href="mailto:ykaul@redhat.com">ykaul@redhat.com</a> <mailto:<a href="mailto:ykaul@redhat.com">ykaul@redhat.com</a>><br>
</span><span class="gmail-">> > <mailto:<a href="mailto:ykaul@redhat.com">ykaul@redhat.com</a> <mailto:<a href="mailto:ykaul@redhat.com">ykaul@redhat.com</a>>>> wrote:<br>
> > ><br>
> > ><br>
> > > On Fri, Mar 24, 2017 at 6:43 PM, Martin Sivak <<a href="mailto:msivak@redhat.com">msivak@redhat.com</a> <mailto:<a href="mailto:msivak@redhat.com">msivak@redhat.com</a>><br>
</span><div><div class="gmail-h5">> > <mailto:<a href="mailto:msivak@redhat.com">msivak@redhat.com</a> <mailto:<a href="mailto:msivak@redhat.com">msivak@redhat.com</a>>>> wrote:<br>
> > >><br>
> > >> > 2: you can have more api gateways (e.g. more apis)<br>
> tailored for<br>
> > every<br>
> > >> > frontend. I don't think we need this - the current API serves<br>
> > us pretty<br>
> > >> > well<br>
> > >> > in every FE Im involved in. The only thing which I miss<br>
> is the data<br>
> > >> > aggregation.<br>
> > >><br>
> > >> So it does not serve us well. Aggregation of data is one<br>
> the usual<br>
> > >> points of using the gateway.<br>
> > >> Yes microservices are affected by this indeed, but so are<br>
> we because<br>
> > >> implementing the aggregation directly in the current engine<br>
> API layer<br>
> > >> is hard.<br>
> > >><br>
> > >> > So I would go back to the original topic of this thread - do<br>
> > some small<br>
> > >> > change which has a chance to be merged to the project and<br>
> helps<br>
> > us where<br>
> > >> > it<br>
> > >> > hurts.<br>
> > ><br>
> > ><br>
> > > I'm wondering if very specific additional REST API calls can<br>
> suffice.<br>
> > > For example, a 'Get VM + disks + NIC' API call seems<br>
> reasonable to<br>
> > add for<br>
> > > the various clients who commonly need it.<br>
> > ><br>
> > >><br>
> > >> Can a simple HTTP/2 to HTTP/AJP gateway be the simplest<br>
> solution? Our<br>
> > >> Apache might even have a module for it already.<br>
> > ><br>
> > ><br>
> > > Current Apache used has only experimental module for it.<br>
> > > Undertow is supposed to have a better support. I wonder<br>
> when/if we<br>
> > can drop<br>
> > > Apache...<br>
> > > Y.<br>
> > ><br>
> > >><br>
> > >> That way you can multiplex all the REST calls using a<br>
> single tcp<br>
> > >> connection (and a single SSL negotiation).<br>
> > >><br>
> > >> A custom SSO enabled service like that might be even better<br>
> as it<br>
> > >> would be able to skip the authentication<br>
> > >> layers too and that would lower the engine load. But I am<br>
> not sure it<br>
> > >> is possible with the current codebase.<br>
> > >><br>
> > >> Martin<br>
> > >><br>
> > >> On Fri, Mar 24, 2017 at 4:22 PM, Tomas Jelinek<br>
> > <<a href="mailto:tjelinek@redhat.com">tjelinek@redhat.com</a> <mailto:<a href="mailto:tjelinek@redhat.com">tjelinek@redhat.com</a>><br>
</div></div>> <mailto:<a href="mailto:tjelinek@redhat.com">tjelinek@redhat.com</a> <mailto:<a href="mailto:tjelinek@redhat.com">tjelinek@redhat.com</a>>>><br>
<span class="gmail-">> > >> wrote:<br>
> > >> ><br>
> > >> ><br>
> > >> > On Fri, Mar 24, 2017 at 3:58 PM, Martin Sivak<br>
> > <<a href="mailto:msivak@redhat.com">msivak@redhat.com</a> <mailto:<a href="mailto:msivak@redhat.com">msivak@redhat.com</a>><br>
</span><div><div class="gmail-h5">> <mailto:<a href="mailto:msivak@redhat.com">msivak@redhat.com</a> <mailto:<a href="mailto:msivak@redhat.com">msivak@redhat.com</a>>>> wrote:<br>
> > >> >><br>
> > >> >> > I feel like every REST API I've ever worked with has<br>
> had the<br>
> > >> >> > aggregation<br>
> > >> >> > +<br>
> > >> >> > projection problem. It's like we're trying to use REST<br>
> as a<br>
> > >> >> > replacement<br>
> > >> >> > for<br>
> > >> >> > SQL -- but the logic that executes the "SQL" lives in a<br>
> > browser now,<br>
> > >> >> > and<br>
> > >> >> > it<br>
> > >> >> > used to live on a server close to the DB. And REST isn't<br>
> > expressive<br>
> > >> >> > for<br>
> > >> >> > selecting data like SQL is.<br>
> > >> >><br>
> > >> >> The current industry solution I know about is called API<br>
> gateway..<br>
> > >> >> most of the big players have internal API with lots of low<br>
> > level stuff<br>
> > >> >> and then couple of external API gateways tailored to<br>
> what the<br>
> > client<br>
> > >> >> needs.<br>
> > >> >><br>
> > >> >> <a href="http://microservices.io/patterns/apigateway.html" rel="noreferrer" target="_blank">http://microservices.io/<wbr>patterns/apigateway.html</a><br>
> <<a href="http://microservices.io/patterns/apigateway.html" rel="noreferrer" target="_blank">http://microservices.io/<wbr>patterns/apigateway.html</a>><br>
> > <<a href="http://microservices.io/patterns/apigateway.html" rel="noreferrer" target="_blank">http://microservices.io/<wbr>patterns/apigateway.html</a><br>
> <<a href="http://microservices.io/patterns/apigateway.html" rel="noreferrer" target="_blank">http://microservices.io/<wbr>patterns/apigateway.html</a>>> (check the backend<br>
> > >> >> for frontend section)<br>
> > >> >><br>
> > >> >> This trend is also visible when you think about services<br>
> that<br>
> > offer<br>
> > >> >> API gateway management and billing like<br>
> > >> >> <a href="https://aws.amazon.com/api-gateway/" rel="noreferrer" target="_blank">https://aws.amazon.com/api-<wbr>gateway/</a><br>
> <<a href="https://aws.amazon.com/api-gateway/" rel="noreferrer" target="_blank">https://aws.amazon.com/api-<wbr>gateway/</a>><br>
> > <<a href="https://aws.amazon.com/api-gateway/" rel="noreferrer" target="_blank">https://aws.amazon.com/api-<wbr>gateway/</a><br>
> <<a href="https://aws.amazon.com/api-gateway/" rel="noreferrer" target="_blank">https://aws.amazon.com/api-<wbr>gateway/</a>>> or our very own<br>
> > >> >> <a href="https://www.3scale.net/" rel="noreferrer" target="_blank">https://www.3scale.net/</a><br>
> > >> ><br>
> > >> ><br>
> > >> > right, but the api gateway solves 2 problems:<br>
> > >> ><br>
> > >> > 1: if you have a microservice architecture it is hard for<br>
> > frontend to<br>
> > >> > talk<br>
> > >> > to 20 different moving services. So the gateway hides this<br>
> > complexity<br>
> > >> > behind<br>
> > >> > it. This is not the problem we have.<br>
> > >> ><br>
> > >> > 2: you can have more api gateways (e.g. more apis)<br>
> tailored for<br>
> > every<br>
> > >> > frontend. I don't think we need this - the current API serves<br>
> > us pretty<br>
> > >> > well<br>
> > >> > in every FE Im involved in. The only thing which I miss<br>
> is the data<br>
> > >> > aggregation.<br>
> > >> ><br>
> > >> > So I would go back to the original topic of this thread - do<br>
> > some small<br>
> > >> > change which has a chance to be merged to the project and<br>
> helps<br>
> > us where<br>
> > >> > it<br>
> > >> > hurts.<br>
> > >> ><br>
> > >> >><br>
> > >> >><br>
> > >> >><br>
> > >> >> Martin<br>
> > >> >><br>
> > >> >> On Fri, Mar 24, 2017 at 3:47 PM, Greg Sheremeta<br>
> > <<a href="mailto:gshereme@redhat.com">gshereme@redhat.com</a> <mailto:<a href="mailto:gshereme@redhat.com">gshereme@redhat.com</a>><br>
</div></div>> <mailto:<a href="mailto:gshereme@redhat.com">gshereme@redhat.com</a> <mailto:<a href="mailto:gshereme@redhat.com">gshereme@redhat.com</a>>>><br>
<span class="gmail-">> > >> >> wrote:<br>
> > >> >> > I feel like every REST API I've ever worked with has had the<br>
> > >> >> > aggregation<br>
> > >> >> > +<br>
> > >> >> > projection problem. It's like we're trying to use REST as a<br>
> > >> >> > replacement<br>
> > >> >> > for<br>
> > >> >> > SQL -- but the logic that executes the "SQL" lives in a<br>
> > browser now,<br>
> > >> >> > and<br>
> > >> >> > it<br>
> > >> >> > used to live on a server close to the DB. And REST isn't<br>
> > expressive<br>
> > >> >> > for<br>
> > >> >> > selecting data like SQL is.<br>
> > >> >> ><br>
> > >> >> > There must be some industry solution to this "I want to do<br>
> > SQL over<br>
> > >> >> > REST"<br>
> > >> >> > problem.<br>
> > >> >> ><br>
> > >> >> > On Fri, Mar 24, 2017 at 5:54 AM, Martin Sivak<br>
> > <<a href="mailto:msivak@redhat.com">msivak@redhat.com</a> <mailto:<a href="mailto:msivak@redhat.com">msivak@redhat.com</a>><br>
</span>> <mailto:<a href="mailto:msivak@redhat.com">msivak@redhat.com</a> <mailto:<a href="mailto:msivak@redhat.com">msivak@redhat.com</a>>>><br>
<div><div class="gmail-h5">> > >> >> > wrote:<br>
> > >> >> >><br>
> > >> >> >> > for quite some time I have been more or less<br>
> involved in<br>
> > >> >> >> > development<br>
> > >> >> >> > of<br>
> > >> >> >> > various UIs for oVirt based entirely on the oVirt's<br>
> REST API<br>
> > >> >> >> > ranging<br>
> > >> >> >> > from<br>
> > >> >> >> > the quite mature moVirt [1] through some cockpit<br>
> > extensions to a<br>
> > >> >> >> > young<br>
> > >> >> >> > and<br>
> > >> >> >> > experimental user portal replacement [2].<br>
> > >> >> >><br>
> > >> >> >> oVirt optimizer has the same issue..<br>
> > >> >> >><br>
> > >> >> >> > 2: add some tiny service which would just accept a<br>
> list of<br>
> > >> >> >> > queries,<br>
> > >> >> >> > execute<br>
> > >> >> >> > them locally (but using real HTTP requests) and<br>
> return in one<br>
> > >> >> >> > bulk. A<br>
> > >> >> >> > naive<br>
> > >> >> >> > implementation just to give a sense of what I mean of<br>
> > this would<br>
> > >> >> >> > be a<br>
> > >> >> >> > shell<br>
> > >> >> >> > script getting list of strings like<br>
> > >> >> >> ><br>
> "<a href="https://localhost/ovirt-engine/api/vms/123/sessions" rel="noreferrer" target="_blank">https://localhost/ovirt-<wbr>engine/api/vms/123/sessions</a><br>
> <<a href="https://localhost/ovirt-engine/api/vms/123/sessions" rel="noreferrer" target="_blank">https://localhost/ovirt-<wbr>engine/api/vms/123/sessions</a>><br>
> > <<a href="https://localhost/ovirt-engine/api/vms/123/sessions" rel="noreferrer" target="_blank">https://localhost/ovirt-<wbr>engine/api/vms/123/sessions</a><br>
> <<a href="https://localhost/ovirt-engine/api/vms/123/sessions" rel="noreferrer" target="_blank">https://localhost/ovirt-<wbr>engine/api/vms/123/sessions</a>>>" iterate over<br>
> > >> >> >> > them<br>
> > >> >> >> > and<br>
> > >> >> >> > do a curl request for each, mangle the results into one<br>
> > string and<br>
> > >> >> >> > return<br>
> > >> >> >> > (credits for this idea to msivak). Easy to implement,<br>
> > possibility<br>
> > >> >> >> > to<br>
> > >> >> >> > add<br>
> > >> >> >> > also projections later to save some bandwidth. But the<br>
> > API would<br>
> > >> >> >> > anyway<br>
> > >> >> >> > be<br>
> > >> >> >> > hammered by bunch of queries, only the network<br>
> roundtrip<br>
> > would be<br>
> > >> >> >> > saved.<br>
> > >> >> >><br>
> > >> >> >> The biggest cost for (especially mobile) clients is<br>
> the cost of<br>
> > >> >> >> establishing new SSL connection. SSL is also pretty<br>
> > expensive on the<br>
> > >> >> >> server side.<br>
> > >> >> >><br>
> > >> >> >> So running the aggregation service on the<br>
> ovirt-engine machine<br>
> > >> >> >> (behind<br>
> > >> >> >> Apache) means the client will do a single SSL request<br>
> with<br>
> > list of N<br>
> > >> >> >> urls and the local "reverse-proxy" will perform single<br>
> > >> >> >> authentication<br>
> > >> >> >> and N plain HTTP requests (or even better - AJP). It<br>
> won't<br>
> > remove<br>
> > >> >> >> any<br>
> > >> >> >> time from the actual command run time, but it will reduce<br>
> > protocol<br>
> > >> >> >> overhead.<br>
> > >> >> >><br>
> > >> >> >> I think this is the simplest first step that requires<br>
> almost no<br>
> > >> >> >> change<br>
> > >> >> >> to existing infrastructure.<br>
> > >> >> >><br>
> > >> >> >> --<br>
> > >> >> >> Martin Sivak<br>
> > >> >> >> SLA / oVirt<br>
> > >> >> >><br>
> > >> >> >> On Fri, Mar 24, 2017 at 10:20 AM, Tomas Jelinek<br>
> > >> >> >> <<a href="mailto:tjelinek@redhat.com">tjelinek@redhat.com</a> <mailto:<a href="mailto:tjelinek@redhat.com">tjelinek@redhat.com</a>><br>
</div></div>> <mailto:<a href="mailto:tjelinek@redhat.com">tjelinek@redhat.com</a> <mailto:<a href="mailto:tjelinek@redhat.com">tjelinek@redhat.com</a>>>><br>
<div><div class="gmail-h5">> > >> >> >> wrote:<br>
> > >> >> >> > Hi All,<br>
> > >> >> >> ><br>
> > >> >> >> > for quite some time I have been more or less<br>
> involved in<br>
> > >> >> >> > development<br>
> > >> >> >> > of<br>
> > >> >> >> > various UIs for oVirt based entirely on the oVirt's<br>
> REST API<br>
> > >> >> >> > ranging<br>
> > >> >> >> > from<br>
> > >> >> >> > the quite mature moVirt [1] through some cockpit<br>
> > extensions to a<br>
> > >> >> >> > young<br>
> > >> >> >> > and<br>
> > >> >> >> > experimental user portal replacement [2].<br>
> > >> >> >> ><br>
> > >> >> >> > One issue we hit over and over again is the missing<br>
> data<br>
> > >> >> >> > aggregation.<br>
> > >> >> >> > In<br>
> > >> >> >> > the<br>
> > >> >> >> > 3.x era we used to use in moVirt the detail=something<br>
> > >> >> >> > api to get the disks and nics of the VM, something<br>
> like:<br>
> > >> >> >> ><br>
> > >> >> >> > GET /ovirt-engine/api/vms<br>
> > >> >> >> > Accept: application/json; detail=disks<br>
> > >> >> >> ><br>
> > >> >> >> > This allowed us to store this data in local database<br>
> > leading to<br>
> > >> >> >> > great<br>
> > >> >> >> > user<br>
> > >> >> >> > experience. Since this feature has been removed in 4.x<br>
> > API [3]<br>
> > >> >> >> > we needed to retire to a different solution. When<br>
> the VM<br>
> > detail is<br>
> > >> >> >> > selected<br>
> > >> >> >> > by the user, start loading the disks and nics and hope<br>
> > the user<br>
> > >> >> >> > will not be fast enough to see the delay. The UX is<br>
> > slightly worse<br>
> > >> >> >> > bug<br>
> > >> >> >> > kinda<br>
> > >> >> >> > acceptable.<br>
> > >> >> >> ><br>
> > >> >> >> > We hit this issue harder in the new user portal [2],<br>
> > because we<br>
> > >> >> >> > already<br>
> > >> >> >> > have<br>
> > >> >> >> > the VM cached and show the whole VM in one screen.<br>
> So, if<br>
> > you pick<br>
> > >> >> >> > it,<br>
> > >> >> >> > you<br>
> > >> >> >> > will get it's details immediately.<br>
> > >> >> >> > But, since you don't have all the details, we need<br>
> to do an<br>
> > >> >> >> > additional<br>
> > >> >> >> > call<br>
> > >> >> >> > (two actually) to load this data and they start to<br>
> appear<br>
> > later.<br>
> > >> >> >> > So, something which would be very fast and smooth<br>
> starts<br>
> > to feel<br>
> > >> >> >> > sluggish.<br>
> > >> >> >> ><br>
> > >> >> >> > Recently, we hit this issue again which forced us to<br>
> > sacrifice the<br>
> > >> >> >> > UX<br>
> > >> >> >> > even<br>
> > >> >> >> > more - it is the "console in use" feature of user<br>
> portal.<br>
> > >> >> >> > The use case is this:<br>
> > >> >> >> > - if the console is already taken by some user,<br>
> there are<br>
> > >> >> >> > complications<br>
> > >> >> >> > if<br>
> > >> >> >> > other current user tryes to take it as well (will avoid<br>
> > details<br>
> > >> >> >> > about<br>
> > >> >> >> > settings and permissins involved, but long story short,<br>
> > the user<br>
> > >> >> >> > will<br>
> > >> >> >> > probably not be allowed to connect to it. The<br>
> "probably"<br>
> > is the<br>
> > >> >> >> > key<br>
> > >> >> >> > here<br>
> > >> >> >> > since we can not do any intelligent decision in<br>
> advance,<br>
> > we can<br>
> > >> >> >> > only<br>
> > >> >> >> > warn<br>
> > >> >> >> > the user that the console is taken).<br>
> > >> >> >> > - in the current GWT user portal, if the VM's<br>
> console is<br>
> > taken, it<br>
> > >> >> >> > is<br>
> > >> >> >> > shown<br>
> > >> >> >> > on the VM's "box" that "console is taken". This was<br>
> a highly<br>
> > >> >> >> > requested<br>
> > >> >> >> > feature<br>
> > >> >> >> > - to get this information using the current REST<br>
> API, we<br>
> > need to<br>
> > >> >> >> > go<br>
> > >> >> >> > to<br>
> > >> >> >> > the<br>
> > >> >> >> > /vms/<vmid>/sessions subcollection. To get this for all<br>
> > VMs, it<br>
> > >> >> >> > would<br>
> > >> >> >> > be<br>
> > >> >> >> > doing N queries per poll which we can not afford<br>
> > >> >> >> > - so the current PR [4] will probably end up to only<br>
> > check it on<br>
> > >> >> >> > the<br>
> > >> >> >> > attempt<br>
> > >> >> >> > to connect to the console warning the user. Maybe<br>
> it will<br>
> > be also<br>
> > >> >> >> > shown<br>
> > >> >> >> > in<br>
> > >> >> >> > Vm details. But the UX in case the user will look for a<br>
> > VM which<br>
> > >> >> >> > has<br>
> > >> >> >> > free<br>
> > >> >> >> > console will suffer significantly (e.g. try one by one<br>
> > until some<br>
> > >> >> >> > opens<br>
> > >> >> >> > or<br>
> > >> >> >> > look at details one by one to see if the warning<br>
> appears<br>
> > (with a<br>
> > >> >> >> > delay))<br>
> > >> >> >> ><br>
> > >> >> >> > I understand that embedding the details of the VM<br>
> to the<br>
> > response<br>
> > >> >> >> > comes<br>
> > >> >> >> > with<br>
> > >> >> >> > a cost, namely:<br>
> > >> >> >> > - performance hit<br>
> > >> >> >> > - complexity of the API code<br>
> > >> >> >> > - the "cleanness" of REST suffers<br>
> > >> >> >> ><br>
> > >> >> >> > But I think we should seriously consider to provide<br>
> some<br>
> > option to<br>
> > >> >> >> > data<br>
> > >> >> >> > aggregation.<br>
> > >> >> >> ><br>
> > >> >> >> > I know this has been discussed many times with no<br>
> result,<br>
> > but I<br>
> > >> >> >> > think<br>
> > >> >> >> > it<br>
> > >> >> >> > is<br>
> > >> >> >> > time to bring this topic up again. I'll try to<br>
> summarize the<br>
> > >> >> >> > (failed)<br>
> > >> >> >> > attempts tried so far:<br>
> > >> >> >> > - the detail=<something> parameter with ad-hoc<br>
> embedding<br>
> > of data.<br>
> > >> >> >> > This<br>
> > >> >> >> > has<br>
> > >> >> >> > been there and removed in 4.0 [3]<br>
> > >> >> >> > - the DoctorREST project - e.g. a proxy above the<br>
> current<br>
> > api. The<br>
> > >> >> >> > idea<br>
> > >> >> >> > was<br>
> > >> >> >> > to create a service which will be independent of<br>
> the engine<br>
> > >> >> >> > itself,<br>
> > >> >> >> > will<br>
> > >> >> >> > locally poll the engine's REST, store all data in local<br>
> > (mongo)DB<br>
> > >> >> >> > and<br>
> > >> >> >> > provide a rich api with aggregations and<br>
> projections and push<br>
> > >> >> >> > notifications.<br>
> > >> >> >> > This polling of everything to get the data to<br>
> DoctorREST<br>
> > proved to<br>
> > >> >> >> > be<br>
> > >> >> >> > pretty<br>
> > >> >> >> > costy, so also a more invasive approach of pushing<br>
> data from<br>
> > >> >> >> > engine<br>
> > >> >> >> > to<br>
> > >> >> >> > doctor has been discused [5]. None of this two<br>
> approaches<br>
> > have<br>
> > >> >> >> > been<br>
> > >> >> >> > accepted<br>
> > >> >> >> > (too complicated, too invasive).<br>
> > >> >> >> > - writing some custom ad-hoc servlet serving only a<br>
> > purpose of one<br>
> > >> >> >> > frontend<br>
> > >> >> >> > - this is actually there for the dashboard, but it<br>
> is not a<br>
> > >> >> >> > generic<br>
> > >> >> >> > solution<br>
> > >> >> >> > for the other frontends and we really should not<br>
> develop<br>
> > custom<br>
> > >> >> >> > "APIs"<br>
> > >> >> >> > for<br>
> > >> >> >> > every frontend<br>
> > >> >> >> > - there were some other proposals discussed (some<br>
> 3th party<br>
> > >> >> >> > solutions<br>
> > >> >> >> > etc)<br>
> > >> >> >> > but I think none of them made it even to a PoC<br>
> > >> >> >> ><br>
> > >> >> >> > So, now I would try again and try small to get at<br>
> least some<br>
> > >> >> >> > benefit.<br>
> > >> >> >> > I<br>
> > >> >> >> > see<br>
> > >> >> >> > 2 paths we could try:<br>
> > >> >> >> > 1: embed something which burns us immediatly, e.g. the<br>
> > /sessions<br>
> > >> >> >> > into<br>
> > >> >> >> > VMs. I<br>
> > >> >> >> > really liked the ;detail=sessions approach, could<br>
> we move<br>
> > it back?<br>
> > >> >> >> > 2: add some tiny service which would just accept a<br>
> list of<br>
> > >> >> >> > queries,<br>
> > >> >> >> > execute<br>
> > >> >> >> > them locally (but using real HTTP requests) and<br>
> return in one<br>
> > >> >> >> > bulk. A<br>
> > >> >> >> > naive<br>
> > >> >> >> > implementation just to give a sense of what I mean of<br>
> > this would<br>
> > >> >> >> > be a<br>
> > >> >> >> > shell<br>
> > >> >> >> > script getting list of strings like<br>
> > >> >> >> ><br>
> "<a href="https://localhost/ovirt-engine/api/vms/123/sessions" rel="noreferrer" target="_blank">https://localhost/ovirt-<wbr>engine/api/vms/123/sessions</a><br>
> <<a href="https://localhost/ovirt-engine/api/vms/123/sessions" rel="noreferrer" target="_blank">https://localhost/ovirt-<wbr>engine/api/vms/123/sessions</a>><br>
> > <<a href="https://localhost/ovirt-engine/api/vms/123/sessions" rel="noreferrer" target="_blank">https://localhost/ovirt-<wbr>engine/api/vms/123/sessions</a><br>
> <<a href="https://localhost/ovirt-engine/api/vms/123/sessions" rel="noreferrer" target="_blank">https://localhost/ovirt-<wbr>engine/api/vms/123/sessions</a>>>" iterate over<br>
> > >> >> >> > them<br>
> > >> >> >> > and<br>
> > >> >> >> > do a curl request for each, mangle the results into one<br>
> > string and<br>
> > >> >> >> > return<br>
> > >> >> >> > (credits for this idea to msivak). Easy to implement,<br>
> > possibility<br>
> > >> >> >> > to<br>
> > >> >> >> > add<br>
> > >> >> >> > also projections later to save some bandwidth. But the<br>
> > API would<br>
> > >> >> >> > anyway<br>
> > >> >> >> > be<br>
> > >> >> >> > hammered by bunch of queries, only the network<br>
> roundtrip<br>
> > would be<br>
> > >> >> >> > saved.<br>
> > >> >> >> > 3: any other simple approaches?<br>
> > >> >> >> ><br>
> > >> >> >> > I honestly prefer the first approach. It is not<br>
> > beautiful, it is<br>
> > >> >> >> > not<br>
> > >> >> >> > REST-ful, but it is easy to implement, very<br>
> pragmatic and<br>
> > useful.<br>
> > >> >> >> > What do you think?<br>
> > >> >> >> ><br>
> > >> >> >> > Thank you and sorry for the long mail :)<br>
> > >> >> >> > Tomas<br>
> > >> >> >> ><br>
> > >> >> >> > [1]: <a href="https://github.com/oVirt/moVirt" rel="noreferrer" target="_blank">https://github.com/oVirt/<wbr>moVirt</a><br>
> <<a href="https://github.com/oVirt/moVirt" rel="noreferrer" target="_blank">https://github.com/oVirt/<wbr>moVirt</a>><br>
> > <<a href="https://github.com/oVirt/moVirt" rel="noreferrer" target="_blank">https://github.com/oVirt/<wbr>moVirt</a><br>
> <<a href="https://github.com/oVirt/moVirt" rel="noreferrer" target="_blank">https://github.com/oVirt/<wbr>moVirt</a>>><br>
> > >> >> >> > [2]: <a href="https://github.com/oVirt/ovirt-web-ui" rel="noreferrer" target="_blank">https://github.com/oVirt/<wbr>ovirt-web-ui</a><br>
> <<a href="https://github.com/oVirt/ovirt-web-ui" rel="noreferrer" target="_blank">https://github.com/oVirt/<wbr>ovirt-web-ui</a>><br>
> > <<a href="https://github.com/oVirt/ovirt-web-ui" rel="noreferrer" target="_blank">https://github.com/oVirt/<wbr>ovirt-web-ui</a><br>
> <<a href="https://github.com/oVirt/ovirt-web-ui" rel="noreferrer" target="_blank">https://github.com/oVirt/<wbr>ovirt-web-ui</a>>><br>
> > >> >> >> > [3]: <a href="https://gerrit.ovirt.org/#/c/61260" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>61260</a><br>
> <<a href="https://gerrit.ovirt.org/#/c/61260" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>61260</a>><br>
> > <<a href="https://gerrit.ovirt.org/#/c/61260" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>61260</a><br>
> <<a href="https://gerrit.ovirt.org/#/c/61260" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>61260</a>>><br>
> > >> >> >> > [4]:<br>
> <a href="https://github.com/oVirt/ovirt-web-ui/pull/106/" rel="noreferrer" target="_blank">https://github.com/oVirt/<wbr>ovirt-web-ui/pull/106/</a><br>
> <<a href="https://github.com/oVirt/ovirt-web-ui/pull/106/" rel="noreferrer" target="_blank">https://github.com/oVirt/<wbr>ovirt-web-ui/pull/106/</a>><br>
> > <<a href="https://github.com/oVirt/ovirt-web-ui/pull/106/" rel="noreferrer" target="_blank">https://github.com/oVirt/<wbr>ovirt-web-ui/pull/106/</a><br>
> <<a href="https://github.com/oVirt/ovirt-web-ui/pull/106/" rel="noreferrer" target="_blank">https://github.com/oVirt/<wbr>ovirt-web-ui/pull/106/</a>>><br>
> > >> >> >> > [5]: <a href="https://gerrit.ovirt.org/#/c/45233/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>45233/</a><br>
> <<a href="https://gerrit.ovirt.org/#/c/45233/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>45233/</a>><br>
> > <<a href="https://gerrit.ovirt.org/#/c/45233/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>45233/</a><br>
> <<a href="https://gerrit.ovirt.org/#/c/45233/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>45233/</a>>><br>
> > >> >> >> ><br>
> > >> >> >> ><br>
> > >> >> >> > ______________________________<wbr>_________________<br>
> > >> >> >> > Devel mailing list<br>
> > >> >> >> > <a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a> <mailto:<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a>><br>
</div></div>> <mailto:<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a> <mailto:<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a>>><br>
<span class="gmail-">> > >> >> >> > <a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
> <<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a>><br>
> > <<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
> <<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a>>><br>
> > >> >> >> ______________________________<wbr>_________________<br>
> > >> >> >> Devel mailing list<br>
> > >> >> >> <a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a> <mailto:<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a>><br>
</span>> <mailto:<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a> <mailto:<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a>>><br>
> > >> >> >> <a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
> <<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a>><br>
<span class="gmail-">> > <<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
> <<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a>>><br>
> > >> >> ><br>
> > >> >> ><br>
> > >> >> ><br>
> > >> >> ><br>
> > >> >> > --<br>
> > >> >> > Greg Sheremeta, MBA<br>
> > >> >> > Red Hat, Inc.<br>
> > >> >> > Sr. Software Engineer<br>
> > >> >> > <a href="mailto:gshereme@redhat.com">gshereme@redhat.com</a> <mailto:<a href="mailto:gshereme@redhat.com">gshereme@redhat.com</a>><br>
</span>> <mailto:<a href="mailto:gshereme@redhat.com">gshereme@redhat.com</a> <mailto:<a href="mailto:gshereme@redhat.com">gshereme@redhat.com</a>>><br>
<span class="gmail-">> > >> ><br>
> > >> ><br>
> > >> ______________________________<wbr>_________________<br>
> > >> Devel mailing list<br>
> > >> <a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a> <mailto:<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a>><br>
</span>> <mailto:<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a> <mailto:<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a>>><br>
> > >> <a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
> <<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a>><br>
> > <<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
> <<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a>>><br>
> > ><br>
> > ><br>
> ><br>
> ><br>
><br>
><br>
<br>
</blockquote></div><br></div></div>