<div dir="ltr"><div>I feel like every REST API I've ever worked with has had the aggregation + projection problem. It's like we're trying to use REST as a replacement for SQL -- but the logic that executes the "SQL" lives in a browser now, and it used to live on a server close to the DB. And REST isn't expressive for selecting data like SQL is.<br></div><div><br></div><div>There must be some industry solution to this "I want to do SQL over REST" problem.</div><div><br></div><div class="gmail_extra"><div class="gmail_quote">On Fri, Mar 24, 2017 at 5:54 AM, Martin Sivak <span dir="ltr"><<a href="mailto:msivak@redhat.com" target="_blank">msivak@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">> for quite some time I have been more or less involved in development of<br>
> various UIs for oVirt based entirely on the oVirt's REST API ranging from<br>
> the quite mature moVirt [1] through some cockpit extensions to a young and<br>
> experimental user portal replacement [2].<br>
<br>
</span>oVirt optimizer has the same issue..<br>
<span class=""><br>
> 2: add some tiny service which would just accept a list of queries, execute<br>
> them locally (but using real HTTP requests) and return in one bulk. A naive<br>
> implementation just to give a sense of what I mean of this would be a shell<br>
> script getting list of strings like<br>
> "<a href="https://localhost/ovirt-engine/api/vms/123/sessions" rel="noreferrer" target="_blank">https://localhost/ovirt-<wbr>engine/api/vms/123/sessions</a>" iterate over them and<br>
> do a curl request for each, mangle the results into one string and return<br>
> (credits for this idea to msivak). Easy to implement, possibility to add<br>
> also projections later to save some bandwidth. But the API would anyway be<br>
> hammered by bunch of queries, only the network roundtrip would be saved.<br>
<br>
</span>The biggest cost for (especially mobile) clients is the cost of<br>
establishing new SSL connection. SSL is also pretty expensive on the<br>
server side.<br>
<br>
So running the aggregation service on the ovirt-engine machine (behind<br>
Apache) means the client will do a single SSL request with list of N<br>
urls and the local "reverse-proxy" will perform single authentication<br>
and N plain HTTP requests (or even better - AJP). It won't remove any<br>
time from the actual command run time, but it will reduce protocol<br>
overhead.<br>
<br>
I think this is the simplest first step that requires almost no change<br>
to existing infrastructure.<br>
<br>
--<br>
Martin Sivak<br>
SLA / oVirt<br>
<div><div class="h5"><br>
On Fri, Mar 24, 2017 at 10:20 AM, Tomas Jelinek <<a href="mailto:tjelinek@redhat.com">tjelinek@redhat.com</a>> wrote:<br>
> Hi All,<br>
><br>
> for quite some time I have been more or less involved in development of<br>
> various UIs for oVirt based entirely on the oVirt's REST API ranging from<br>
> the quite mature moVirt [1] through some cockpit extensions to a young and<br>
> experimental user portal replacement [2].<br>
><br>
> One issue we hit over and over again is the missing data aggregation. In the<br>
> 3.x era we used to use in moVirt the detail=something<br>
> api to get the disks and nics of the VM, something like:<br>
><br>
> GET /ovirt-engine/api/vms<br>
> Accept: application/json; detail=disks<br>
><br>
> This allowed us to store this data in local database leading to great user<br>
> experience. Since this feature has been removed in 4.x API [3]<br>
> we needed to retire to a different solution. When the VM detail is selected<br>
> by the user, start loading the disks and nics and hope the user<br>
> will not be fast enough to see the delay. The UX is slightly worse bug kinda<br>
> acceptable.<br>
><br>
> We hit this issue harder in the new user portal [2], because we already have<br>
> the VM cached and show the whole VM in one screen. So, if you pick it, you<br>
> will get it's details immediately.<br>
> But, since you don't have all the details, we need to do an additional call<br>
> (two actually) to load this data and they start to appear later.<br>
> So, something which would be very fast and smooth starts to feel sluggish.<br>
><br>
> Recently, we hit this issue again which forced us to sacrifice the UX even<br>
> more - it is the "console in use" feature of user portal.<br>
> The use case is this:<br>
> - if the console is already taken by some user, there are complications if<br>
> other current user tryes to take it as well (will avoid details about<br>
> settings and permissins involved, but long story short, the user will<br>
> probably not be allowed to connect to it. The "probably" is the key here<br>
> since we can not do any intelligent decision in advance, we can only warn<br>
> the user that the console is taken).<br>
> - in the current GWT user portal, if the VM's console is taken, it is shown<br>
> on the VM's "box" that "console is taken". This was a highly requested<br>
> feature<br>
> - to get this information using the current REST API, we need to go to the<br>
> /vms/<vmid>/sessions subcollection. To get this for all VMs, it would be<br>
> doing N queries per poll which we can not afford<br>
> - so the current PR [4] will probably end up to only check it on the attempt<br>
> to connect to the console warning the user. Maybe it will be also shown in<br>
> Vm details. But the UX in case the user will look for a VM which has free<br>
> console will suffer significantly (e.g. try one by one until some opens or<br>
> look at details one by one to see if the warning appears (with a delay))<br>
><br>
> I understand that embedding the details of the VM to the response comes with<br>
> a cost, namely:<br>
> - performance hit<br>
> - complexity of the API code<br>
> - the "cleanness" of REST suffers<br>
><br>
> But I think we should seriously consider to provide some option to data<br>
> aggregation.<br>
><br>
> I know this has been discussed many times with no result, but I think it is<br>
> time to bring this topic up again. I'll try to summarize the (failed)<br>
> attempts tried so far:<br>
> - the detail=<something> parameter with ad-hoc embedding of data. This has<br>
> been there and removed in 4.0 [3]<br>
> - the DoctorREST project - e.g. a proxy above the current api. The idea was<br>
> to create a service which will be independent of the engine itself, will<br>
> locally poll the engine's REST, store all data in local (mongo)DB and<br>
> provide a rich api with aggregations and projections and push notifications.<br>
> This polling of everything to get the data to DoctorREST proved to be pretty<br>
> costy, so also a more invasive approach of pushing data from engine to<br>
> doctor has been discused [5]. None of this two approaches have been accepted<br>
> (too complicated, too invasive).<br>
> - writing some custom ad-hoc servlet serving only a purpose of one frontend<br>
> - this is actually there for the dashboard, but it is not a generic solution<br>
> for the other frontends and we really should not develop custom "APIs" for<br>
> every frontend<br>
> - there were some other proposals discussed (some 3th party solutions etc)<br>
> but I think none of them made it even to a PoC<br>
><br>
> So, now I would try again and try small to get at least some benefit. I see<br>
> 2 paths we could try:<br>
> 1: embed something which burns us immediatly, e.g. the /sessions into VMs. I<br>
> really liked the ;detail=sessions approach, could we move it back?<br>
> 2: add some tiny service which would just accept a list of queries, execute<br>
> them locally (but using real HTTP requests) and return in one bulk. A naive<br>
> implementation just to give a sense of what I mean of this would be a shell<br>
> script getting list of strings like<br>
> "<a href="https://localhost/ovirt-engine/api/vms/123/sessions" rel="noreferrer" target="_blank">https://localhost/ovirt-<wbr>engine/api/vms/123/sessions</a>" iterate over them and<br>
> do a curl request for each, mangle the results into one string and return<br>
> (credits for this idea to msivak). Easy to implement, possibility to add<br>
> also projections later to save some bandwidth. But the API would anyway be<br>
> hammered by bunch of queries, only the network roundtrip would be saved.<br>
> 3: any other simple approaches?<br>
><br>
> I honestly prefer the first approach. It is not beautiful, it is not<br>
> REST-ful, but it is easy to implement, very pragmatic and useful.<br>
> What do you think?<br>
><br>
> Thank you and sorry for the long mail :)<br>
> Tomas<br>
><br>
> [1]: <a href="https://github.com/oVirt/moVirt" rel="noreferrer" target="_blank">https://github.com/oVirt/<wbr>moVirt</a><br>
> [2]: <a href="https://github.com/oVirt/ovirt-web-ui" rel="noreferrer" target="_blank">https://github.com/oVirt/<wbr>ovirt-web-ui</a><br>
> [3]: <a href="https://gerrit.ovirt.org/#/c/61260" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>61260</a><br>
> [4]: <a href="https://github.com/oVirt/ovirt-web-ui/pull/106/" rel="noreferrer" target="_blank">https://github.com/oVirt/<wbr>ovirt-web-ui/pull/106/</a><br>
> [5]: <a href="https://gerrit.ovirt.org/#/c/45233/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>45233/</a><br>
><br>
><br>
</div></div>> ______________________________<wbr>_________________<br>
> Devel mailing list<br>
> <a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Greg Sheremeta, MBA<br>Red Hat, Inc.<br>Sr. Software Engineer<br><a href="mailto:gshereme@redhat.com" target="_blank">gshereme@redhat.com</a><br></div></div></div>
</div></div>