[ovirt-devel] REST API data aggregation
Martin Betak
mbetak at redhat.com
Mon Mar 27 15:43:52 UTC 2017
On Mon, Mar 27, 2017 at 4:49 PM, Greg Sheremeta <gshereme at redhat.com> wrote:
> On Mon, Mar 27, 2017 at 8:59 AM, Tomas Jelinek <tjelinek at redhat.com>
> wrote:
>
>>
>>
>> On Mon, Mar 27, 2017 at 1:32 PM, Juan Hernández <jhernand at redhat.com>
>> wrote:
>>
>>> On 03/27/2017 01:03 PM, Tomas Jelinek wrote:
>>> >
>>> >
>>> > On Mon, Mar 27, 2017 at 11:21 AM, Juan Hernández <jhernand at redhat.com
>>> > <mailto:jhernand at redhat.com>> wrote:
>>> >
>>> > Top posting, sorry.
>>> >
>>> > There are a few things I'd like to clarify, regarding this subject:
>>> >
>>> > 1. Data aggregation, as requested now by Tomas, and by other
>>> people in
>>> > the past.
>>> >
>>> > We used to have that 'detail' parameter, to aggregate certain very
>>> > specific types of data, in particular to aggregate VM disks and
>>> NICs. We
>>> > removed that in version 4 of the API because the implementation was
>>> > extremely inefficient, from the engine point of view. An innocent
>>> > request like this:
>>> >
>>> > GET /ovirt-engine/api/vms?detail=+disks,+nics
>>> >
>>> > Would generate, with the implementation we used to have, 1 query
>>> for the
>>> > VMs and then as many queries for disks and NICs as VMs in the
>>> system. In
>>> > our scale test environments, for example, with approx 4000 VMs and
>>> 10000
>>> > disks, that would take more than 20 hours to execute.
>>> >
>>> > In addition, we didn't have in the past any mechanism to make this
>>> > available in a generic one, because there was no knowledge in the
>>> API of
>>> > what are 'details'.
>>> >
>>> > In version 4 of the API we introduced a formal (kind of)
>>> specification
>>> > of the API (a.k.a. the model), and int includes knowledge about
>>> what are
>>> > 'links'. For example, the specification of the VM type contains
>>> this:
>>> >
>>> > @Link DiskAttachment[] diskAttachments();
>>> > @Link Nic[] nics();
>>> >
>>> > With this information we are now in a position where we can
>>> implement
>>> > this in a generic way.
>>> >
>>> > We intend to implement this using a mechanism similar to the
>>> existing
>>> > 'detail' parameter:
>>> >
>>> > GET /ovirt-engine/api/vms/123?follow=disk_attachments,nics
>>> >
>>> > The naive implementation of this is to let the API call itself. For
>>> > example, when the user requests to follow the 'disk_attachments'
>>> detail
>>> > the API can just call itself to get that:
>>> >
>>> > GET /ovirt-engine/api/vms/123/disk_attachments
>>> >
>>> > However, we can't use that naive approach, if we do we end with the
>>> > 1+C*N query problem described before. We need to use specific
>>> > implementations for certain frequent use cases, like
>>> VMs+disks+nics, and
>>> > that needs work in the API and in the backend.
>>> >
>>> > Tomas, if you want to help moving this forward, please open a RFE
>>> and
>>> > makes sure it gets attention.
>>>
>>
>> ok, opened: https://bugzilla.redhat.com/show_bug.cgi?id=1436206
>> Will try to get it done soon.
>>
>
> Forgive me if this is radical, but has anyone thought of / discussed using
> a NoSQL alternative to our very normalized SQL db as a way to avoid the
> problem of aggregating details? Using mongodb as an example, embed some of
> the smaller objects, and there's no cost of aggregation there. IIRC, Doctor
> REST uses mongo under the hood.
>
> https://docs.mongodb.com/manual/tutorial/model-embedded-one-to-many-
> relationships-between-documents/
>
Greg, I really appreciate your enthusiasm for NoSQL technologies but we
have to distinguish between the functional requirements for cache of
schemaless frontend-optimized projections (as in the Dr. Rest case) and the
main application data store, supporting transactional business logic. I
don't believe the current architecture could survive the weak
eventual-consistency guarantees of not fully transactional DB (such as
Postgres is) :-)
Best regards,
Martin
>
>
> Greg
>
>
>>
>>
>>> >
>>> >
>>> > This sounds pretty good! I will open, but since we are talking already
>>> > here I'll just use the opportunity to clarify the topic more and than
>>> > I'll open the BZ.
>>> >
>>> > What I can imagine is the GetAllVmsQuery will accept in params also the
>>> > list of details it should provide. Than, the GetAllVmsQuery will
>>> > implement the efficient way of retrieving this info as well.
>>> >
>>> > So, from the API perspective, it will be about taking the
>>> > ?follow=<something> part and passing it to the backend query params.
>>> >
>>> > What you think?
>>> >
>>>
>>> Exactly, that is the point! The API by itself can't optimize database
>>> queries, all it can do is call the backend. It is the backend that has
>>> the opportunity and possibility to send optimized queries to the
>>> database.
>>>
>>> For other less common things we can use the naive approach, and
>>> implement the aggregation in the API itself. But for common use cases,
>>> like VM+disks+nics, we need to do it in an efficient way.
>>>
>>> >
>>> >
>>> > 2. Reuse of TLS sessions.
>>> >
>>> > The part of creating TLS sessions that is expensive is the
>>> generation of
>>> > the shared session key. That can be avoided if both the server and
>>> the
>>> > client are careful and reuse the session, using the session cache
>>> > mechanism built-in into TLS itself. The web servers that we use
>>> (Apache
>>> > and Undertow) do implement this mechanism, and so do most of our
>>> > clients. Make sure that your client uses it as well. In Java this
>>> is
>>> > achieved re-using the SSLContext. We already do that for the
>>> engine to
>>> > VDSM communciation for example. In JavaScript the browser already
>>> takes
>>> > care of this.
>>> >
>>> > 3. Parallelism and latency.
>>> >
>>> > A typical problem that we have is that we send many request to the
>>> > server. For example, to retrieve user sessions for a set of VMs we
>>> tend
>>> > to send many requests like this:
>>> >
>>> > GET /ovirt-engine/api/vms/1/sessions
>>> > GET /ovirt/engine/api/vms/2/sessions
>>> > GET /ovirt-engine/api/vms/3/sessions
>>> > ...
>>> >
>>> > And we do that in a synchronous way: send one, wait for the
>>> result, send
>>> > another one, wait for the result, etc. This means that we don't
>>> take
>>> > advantage of the parallelism of the server and that we add to each
>>> > request the network round trip time. So if we have N requests, we
>>> have
>>> > to wait at least N*RTT.
>>> >
>>> > The web servers that we use support multiple connections, and the
>>> > protocol that we use, HTTP, supports pipe-lining. This means that
>>> you
>>> > can send multiple requests in parallel, and that you can send
>>> multiple
>>> > requests without waiting for the response. To give you an idea of
>>> the
>>> > improvement that can be achieved, we recently added asynchronous
>>> request
>>> > support to the Ruby SDK, with multiple connections and
>>> pipe-lining. In
>>> > our scale testing environment that reduced the time to collect a
>>> > complete inventory from approx 30 min to approx 2 min. Here you
>>> have an
>>> > example:
>>> >
>>> >
>>> > https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/
>>> sdk/examples/asynchronous_inventory.rb
>>> > <https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master
>>> /sdk/examples/asynchronous_inventory.rb>
>>> >
>>> > So make sure that you take advantage of that in your clients. Sadly
>>> > pipe-lining is disabled by default in most browsers, so this isn't
>>> > helpful for JavaScript applications.
>>> >
>>> >
>>> > But we can try what we can do in moVirt about
>>> > this: https://github.com/oVirt/moVirt/issues/260
>>> >
>>>
>>> Sure, I think there are plenty of asynchronous HTTP clients for Android,
>>> worth trying one of them. If you are brave enough you can even consider
>>> using the same library used in the Ruby SDK: libcurl. A bit of JNI here
>>> and there, and you are done.
>>>
>>> >
>>> >
>>> > 4. HTTP/2 support.
>>> >
>>> > The application server that we use, WildFly, supports HTTP/2,
>>> including
>>> > ALPN, out of the box, since version 10.1. We need a mechanism to
>>> > enable it:
>>> >
>>> > core: Add support for enabling HTTP/2
>>> > https://gerrit.ovirt.org/74621
>>> >
>>> > And then we need to get Apache out of the way, for API traffic, at
>>> > least. I think that is something we can do in the context of the
>>> engine
>>> > "podification" effort.
>>> >
>>> > However, note that HTTP/2 won't have that big impact in
>>> performance for
>>> > applications that continue to use a synchronous/serial style of
>>> > interaction with the API.
>>> >
>>> > On 03/24/2017 11:16 PM, Yaniv Kaul wrote:
>>> > >
>>> > >
>>> > > On Fri, Mar 24, 2017 at 8:57 PM, Martin Sivak <msivak at redhat.com
>>> <mailto:msivak at redhat.com>
>>> > > <mailto:msivak at redhat.com <mailto:msivak at redhat.com>>> wrote:
>>> > >
>>> > > > Current Apache used has only experimental module for it.
>>> > > > Undertow is supposed to have a better support. I wonder
>>> when/if we can drop
>>> > > > Apache...
>>> > >
>>> > > The last info I have about that from mperina is that we need
>>> Apache
>>> > > for kerberos support atm.
>>> > >
>>> > >
>>> > > I don't think we need it - I remember reading that Undertow does
>>> support
>>> > > it as well.
>>> > > The only issue is that there are probably 10 people in the world
>>> who
>>> > > know how to configure Undertow for Kerberos, while many do for
>>> Apache.
>>> > > And since we leave it for the user to configure...
>>> > > Y.
>>> > >
>>> > >
>>> > >
>>> > > Martin
>>> > >
>>> > > On Fri, Mar 24, 2017 at 5:30 PM, Yaniv Kaul <
>>> ykaul at redhat.com <mailto:ykaul at redhat.com>
>>> > > <mailto:ykaul at redhat.com <mailto:ykaul at redhat.com>>> wrote:
>>> > > >
>>> > > >
>>> > > > On Fri, Mar 24, 2017 at 6:43 PM, Martin Sivak <
>>> msivak at redhat.com <mailto:msivak at redhat.com>
>>> > > <mailto:msivak at redhat.com <mailto:msivak at redhat.com>>>
>>> wrote:
>>> > > >>
>>> > > >> > 2: you can have more api gateways (e.g. more apis)
>>> > tailored for
>>> > > every
>>> > > >> > frontend. I don't think we need this - the current API
>>> serves
>>> > > us pretty
>>> > > >> > well
>>> > > >> > in every FE Im involved in. The only thing which I miss
>>> > is the data
>>> > > >> > aggregation.
>>> > > >>
>>> > > >> So it does not serve us well. Aggregation of data is one
>>> > the usual
>>> > > >> points of using the gateway.
>>> > > >> Yes microservices are affected by this indeed, but so are
>>> > we because
>>> > > >> implementing the aggregation directly in the current
>>> engine
>>> > API layer
>>> > > >> is hard.
>>> > > >>
>>> > > >> > So I would go back to the original topic of this thread
>>> - do
>>> > > some small
>>> > > >> > change which has a chance to be merged to the project
>>> and
>>> > helps
>>> > > us where
>>> > > >> > it
>>> > > >> > hurts.
>>> > > >
>>> > > >
>>> > > > I'm wondering if very specific additional REST API calls
>>> can
>>> > suffice.
>>> > > > For example, a 'Get VM + disks + NIC' API call seems
>>> > reasonable to
>>> > > add for
>>> > > > the various clients who commonly need it.
>>> > > >
>>> > > >>
>>> > > >> Can a simple HTTP/2 to HTTP/AJP gateway be the simplest
>>> > solution? Our
>>> > > >> Apache might even have a module for it already.
>>> > > >
>>> > > >
>>> > > > Current Apache used has only experimental module for it.
>>> > > > Undertow is supposed to have a better support. I wonder
>>> > when/if we
>>> > > can drop
>>> > > > Apache...
>>> > > > Y.
>>> > > >
>>> > > >>
>>> > > >> That way you can multiplex all the REST calls using a
>>> > single tcp
>>> > > >> connection (and a single SSL negotiation).
>>> > > >>
>>> > > >> A custom SSO enabled service like that might be even
>>> better
>>> > as it
>>> > > >> would be able to skip the authentication
>>> > > >> layers too and that would lower the engine load. But I am
>>> > not sure it
>>> > > >> is possible with the current codebase.
>>> > > >>
>>> > > >> Martin
>>> > > >>
>>> > > >> On Fri, Mar 24, 2017 at 4:22 PM, Tomas Jelinek
>>> > > <tjelinek at redhat.com <mailto:tjelinek at redhat.com>
>>> > <mailto:tjelinek at redhat.com <mailto:tjelinek at redhat.com>>>
>>> > > >> wrote:
>>> > > >> >
>>> > > >> >
>>> > > >> > On Fri, Mar 24, 2017 at 3:58 PM, Martin Sivak
>>> > > <msivak at redhat.com <mailto:msivak at redhat.com>
>>> > <mailto:msivak at redhat.com <mailto:msivak at redhat.com>>> wrote:
>>> > > >> >>
>>> > > >> >> > I feel like every REST API I've ever worked with has
>>> > had the
>>> > > >> >> > aggregation
>>> > > >> >> > +
>>> > > >> >> > projection problem. It's like we're trying to use
>>> REST
>>> > as a
>>> > > >> >> > replacement
>>> > > >> >> > for
>>> > > >> >> > SQL -- but the logic that executes the "SQL" lives
>>> in a
>>> > > browser now,
>>> > > >> >> > and
>>> > > >> >> > it
>>> > > >> >> > used to live on a server close to the DB. And REST
>>> isn't
>>> > > expressive
>>> > > >> >> > for
>>> > > >> >> > selecting data like SQL is.
>>> > > >> >>
>>> > > >> >> The current industry solution I know about is called
>>> API
>>> > gateway..
>>> > > >> >> most of the big players have internal API with lots of
>>> low
>>> > > level stuff
>>> > > >> >> and then couple of external API gateways tailored to
>>> > what the
>>> > > client
>>> > > >> >> needs.
>>> > > >> >>
>>> > > >> >> http://microservices.io/patterns/apigateway.html
>>> > <http://microservices.io/patterns/apigateway.html>
>>> > > <http://microservices.io/patterns/apigateway.html
>>> > <http://microservices.io/patterns/apigateway.html>> (check the
>>> backend
>>> > > >> >> for frontend section)
>>> > > >> >>
>>> > > >> >> This trend is also visible when you think about
>>> services
>>> > that
>>> > > offer
>>> > > >> >> API gateway management and billing like
>>> > > >> >> https://aws.amazon.com/api-gateway/
>>> > <https://aws.amazon.com/api-gateway/>
>>> > > <https://aws.amazon.com/api-gateway/
>>> > <https://aws.amazon.com/api-gateway/>> or our very own
>>> > > >> >> https://www.3scale.net/
>>> > > >> >
>>> > > >> >
>>> > > >> > right, but the api gateway solves 2 problems:
>>> > > >> >
>>> > > >> > 1: if you have a microservice architecture it is hard
>>> for
>>> > > frontend to
>>> > > >> > talk
>>> > > >> > to 20 different moving services. So the gateway hides
>>> this
>>> > > complexity
>>> > > >> > behind
>>> > > >> > it. This is not the problem we have.
>>> > > >> >
>>> > > >> > 2: you can have more api gateways (e.g. more apis)
>>> > tailored for
>>> > > every
>>> > > >> > frontend. I don't think we need this - the current API
>>> serves
>>> > > us pretty
>>> > > >> > well
>>> > > >> > in every FE Im involved in. The only thing which I miss
>>> > is the data
>>> > > >> > aggregation.
>>> > > >> >
>>> > > >> > So I would go back to the original topic of this thread
>>> - do
>>> > > some small
>>> > > >> > change which has a chance to be merged to the project
>>> and
>>> > helps
>>> > > us where
>>> > > >> > it
>>> > > >> > hurts.
>>> > > >> >
>>> > > >> >>
>>> > > >> >>
>>> > > >> >>
>>> > > >> >> Martin
>>> > > >> >>
>>> > > >> >> On Fri, Mar 24, 2017 at 3:47 PM, Greg Sheremeta
>>> > > <gshereme at redhat.com <mailto:gshereme at redhat.com>
>>> > <mailto:gshereme at redhat.com <mailto:gshereme at redhat.com>>>
>>> > > >> >> wrote:
>>> > > >> >> > I feel like every REST API I've ever worked with has
>>> had the
>>> > > >> >> > aggregation
>>> > > >> >> > +
>>> > > >> >> > projection problem. It's like we're trying to use
>>> REST as a
>>> > > >> >> > replacement
>>> > > >> >> > for
>>> > > >> >> > SQL -- but the logic that executes the "SQL" lives
>>> in a
>>> > > browser now,
>>> > > >> >> > and
>>> > > >> >> > it
>>> > > >> >> > used to live on a server close to the DB. And REST
>>> isn't
>>> > > expressive
>>> > > >> >> > for
>>> > > >> >> > selecting data like SQL is.
>>> > > >> >> >
>>> > > >> >> > There must be some industry solution to this "I want
>>> to do
>>> > > SQL over
>>> > > >> >> > REST"
>>> > > >> >> > problem.
>>> > > >> >> >
>>> > > >> >> > On Fri, Mar 24, 2017 at 5:54 AM, Martin Sivak
>>> > > <msivak at redhat.com <mailto:msivak at redhat.com>
>>> > <mailto:msivak at redhat.com <mailto:msivak at redhat.com>>>
>>> > > >> >> > wrote:
>>> > > >> >> >>
>>> > > >> >> >> > for quite some time I have been more or less
>>> > involved in
>>> > > >> >> >> > development
>>> > > >> >> >> > of
>>> > > >> >> >> > various UIs for oVirt based entirely on the
>>> oVirt's
>>> > REST API
>>> > > >> >> >> > ranging
>>> > > >> >> >> > from
>>> > > >> >> >> > the quite mature moVirt [1] through some cockpit
>>> > > extensions to a
>>> > > >> >> >> > young
>>> > > >> >> >> > and
>>> > > >> >> >> > experimental user portal replacement [2].
>>> > > >> >> >>
>>> > > >> >> >> oVirt optimizer has the same issue..
>>> > > >> >> >>
>>> > > >> >> >> > 2: add some tiny service which would just accept a
>>> > list of
>>> > > >> >> >> > queries,
>>> > > >> >> >> > execute
>>> > > >> >> >> > them locally (but using real HTTP requests) and
>>> > return in one
>>> > > >> >> >> > bulk. A
>>> > > >> >> >> > naive
>>> > > >> >> >> > implementation just to give a sense of what I
>>> mean of
>>> > > this would
>>> > > >> >> >> > be a
>>> > > >> >> >> > shell
>>> > > >> >> >> > script getting list of strings like
>>> > > >> >> >> >
>>> > "https://localhost/ovirt-engine/api/vms/123/sessions
>>> > <https://localhost/ovirt-engine/api/vms/123/sessions>
>>> > > <https://localhost/ovirt-engine/api/vms/123/sessions
>>> > <https://localhost/ovirt-engine/api/vms/123/sessions>>" iterate
>>> over
>>> > > >> >> >> > them
>>> > > >> >> >> > and
>>> > > >> >> >> > do a curl request for each, mangle the results
>>> into one
>>> > > string and
>>> > > >> >> >> > return
>>> > > >> >> >> > (credits for this idea to msivak). Easy to
>>> implement,
>>> > > possibility
>>> > > >> >> >> > to
>>> > > >> >> >> > add
>>> > > >> >> >> > also projections later to save some bandwidth.
>>> But the
>>> > > API would
>>> > > >> >> >> > anyway
>>> > > >> >> >> > be
>>> > > >> >> >> > hammered by bunch of queries, only the network
>>> > roundtrip
>>> > > would be
>>> > > >> >> >> > saved.
>>> > > >> >> >>
>>> > > >> >> >> The biggest cost for (especially mobile) clients is
>>> > the cost of
>>> > > >> >> >> establishing new SSL connection. SSL is also pretty
>>> > > expensive on the
>>> > > >> >> >> server side.
>>> > > >> >> >>
>>> > > >> >> >> So running the aggregation service on the
>>> > ovirt-engine machine
>>> > > >> >> >> (behind
>>> > > >> >> >> Apache) means the client will do a single SSL
>>> request
>>> > with
>>> > > list of N
>>> > > >> >> >> urls and the local "reverse-proxy" will perform
>>> single
>>> > > >> >> >> authentication
>>> > > >> >> >> and N plain HTTP requests (or even better - AJP). It
>>> > won't
>>> > > remove
>>> > > >> >> >> any
>>> > > >> >> >> time from the actual command run time, but it will
>>> reduce
>>> > > protocol
>>> > > >> >> >> overhead.
>>> > > >> >> >>
>>> > > >> >> >> I think this is the simplest first step that
>>> requires
>>> > almost no
>>> > > >> >> >> change
>>> > > >> >> >> to existing infrastructure.
>>> > > >> >> >>
>>> > > >> >> >> --
>>> > > >> >> >> Martin Sivak
>>> > > >> >> >> SLA / oVirt
>>> > > >> >> >>
>>> > > >> >> >> On Fri, Mar 24, 2017 at 10:20 AM, Tomas Jelinek
>>> > > >> >> >> <tjelinek at redhat.com <mailto:tjelinek at redhat.com>
>>> > <mailto:tjelinek at redhat.com <mailto:tjelinek at redhat.com>>>
>>> > > >> >> >> wrote:
>>> > > >> >> >> > Hi All,
>>> > > >> >> >> >
>>> > > >> >> >> > for quite some time I have been more or less
>>> > involved in
>>> > > >> >> >> > development
>>> > > >> >> >> > of
>>> > > >> >> >> > various UIs for oVirt based entirely on the
>>> oVirt's
>>> > REST API
>>> > > >> >> >> > ranging
>>> > > >> >> >> > from
>>> > > >> >> >> > the quite mature moVirt [1] through some cockpit
>>> > > extensions to a
>>> > > >> >> >> > young
>>> > > >> >> >> > and
>>> > > >> >> >> > experimental user portal replacement [2].
>>> > > >> >> >> >
>>> > > >> >> >> > One issue we hit over and over again is the
>>> missing
>>> > data
>>> > > >> >> >> > aggregation.
>>> > > >> >> >> > In
>>> > > >> >> >> > the
>>> > > >> >> >> > 3.x era we used to use in moVirt the
>>> detail=something
>>> > > >> >> >> > api to get the disks and nics of the VM, something
>>> > like:
>>> > > >> >> >> >
>>> > > >> >> >> > GET /ovirt-engine/api/vms
>>> > > >> >> >> > Accept: application/json; detail=disks
>>> > > >> >> >> >
>>> > > >> >> >> > This allowed us to store this data in local
>>> database
>>> > > leading to
>>> > > >> >> >> > great
>>> > > >> >> >> > user
>>> > > >> >> >> > experience. Since this feature has been removed
>>> in 4.x
>>> > > API [3]
>>> > > >> >> >> > we needed to retire to a different solution. When
>>> > the VM
>>> > > detail is
>>> > > >> >> >> > selected
>>> > > >> >> >> > by the user, start loading the disks and nics and
>>> hope
>>> > > the user
>>> > > >> >> >> > will not be fast enough to see the delay. The UX
>>> is
>>> > > slightly worse
>>> > > >> >> >> > bug
>>> > > >> >> >> > kinda
>>> > > >> >> >> > acceptable.
>>> > > >> >> >> >
>>> > > >> >> >> > We hit this issue harder in the new user portal
>>> [2],
>>> > > because we
>>> > > >> >> >> > already
>>> > > >> >> >> > have
>>> > > >> >> >> > the VM cached and show the whole VM in one screen.
>>> > So, if
>>> > > you pick
>>> > > >> >> >> > it,
>>> > > >> >> >> > you
>>> > > >> >> >> > will get it's details immediately.
>>> > > >> >> >> > But, since you don't have all the details, we need
>>> > to do an
>>> > > >> >> >> > additional
>>> > > >> >> >> > call
>>> > > >> >> >> > (two actually) to load this data and they start to
>>> > appear
>>> > > later.
>>> > > >> >> >> > So, something which would be very fast and smooth
>>> > starts
>>> > > to feel
>>> > > >> >> >> > sluggish.
>>> > > >> >> >> >
>>> > > >> >> >> > Recently, we hit this issue again which forced us
>>> to
>>> > > sacrifice the
>>> > > >> >> >> > UX
>>> > > >> >> >> > even
>>> > > >> >> >> > more - it is the "console in use" feature of user
>>> > portal.
>>> > > >> >> >> > The use case is this:
>>> > > >> >> >> > - if the console is already taken by some user,
>>> > there are
>>> > > >> >> >> > complications
>>> > > >> >> >> > if
>>> > > >> >> >> > other current user tryes to take it as well (will
>>> avoid
>>> > > details
>>> > > >> >> >> > about
>>> > > >> >> >> > settings and permissins involved, but long story
>>> short,
>>> > > the user
>>> > > >> >> >> > will
>>> > > >> >> >> > probably not be allowed to connect to it. The
>>> > "probably"
>>> > > is the
>>> > > >> >> >> > key
>>> > > >> >> >> > here
>>> > > >> >> >> > since we can not do any intelligent decision in
>>> > advance,
>>> > > we can
>>> > > >> >> >> > only
>>> > > >> >> >> > warn
>>> > > >> >> >> > the user that the console is taken).
>>> > > >> >> >> > - in the current GWT user portal, if the VM's
>>> > console is
>>> > > taken, it
>>> > > >> >> >> > is
>>> > > >> >> >> > shown
>>> > > >> >> >> > on the VM's "box" that "console is taken". This
>>> was
>>> > a highly
>>> > > >> >> >> > requested
>>> > > >> >> >> > feature
>>> > > >> >> >> > - to get this information using the current REST
>>> > API, we
>>> > > need to
>>> > > >> >> >> > go
>>> > > >> >> >> > to
>>> > > >> >> >> > the
>>> > > >> >> >> > /vms/<vmid>/sessions subcollection. To get this
>>> for all
>>> > > VMs, it
>>> > > >> >> >> > would
>>> > > >> >> >> > be
>>> > > >> >> >> > doing N queries per poll which we can not afford
>>> > > >> >> >> > - so the current PR [4] will probably end up to
>>> only
>>> > > check it on
>>> > > >> >> >> > the
>>> > > >> >> >> > attempt
>>> > > >> >> >> > to connect to the console warning the user. Maybe
>>> > it will
>>> > > be also
>>> > > >> >> >> > shown
>>> > > >> >> >> > in
>>> > > >> >> >> > Vm details. But the UX in case the user will look
>>> for a
>>> > > VM which
>>> > > >> >> >> > has
>>> > > >> >> >> > free
>>> > > >> >> >> > console will suffer significantly (e.g. try one
>>> by one
>>> > > until some
>>> > > >> >> >> > opens
>>> > > >> >> >> > or
>>> > > >> >> >> > look at details one by one to see if the warning
>>> > appears
>>> > > (with a
>>> > > >> >> >> > delay))
>>> > > >> >> >> >
>>> > > >> >> >> > I understand that embedding the details of the VM
>>> > to the
>>> > > response
>>> > > >> >> >> > comes
>>> > > >> >> >> > with
>>> > > >> >> >> > a cost, namely:
>>> > > >> >> >> > - performance hit
>>> > > >> >> >> > - complexity of the API code
>>> > > >> >> >> > - the "cleanness" of REST suffers
>>> > > >> >> >> >
>>> > > >> >> >> > But I think we should seriously consider to
>>> provide
>>> > some
>>> > > option to
>>> > > >> >> >> > data
>>> > > >> >> >> > aggregation.
>>> > > >> >> >> >
>>> > > >> >> >> > I know this has been discussed many times with no
>>> > result,
>>> > > but I
>>> > > >> >> >> > think
>>> > > >> >> >> > it
>>> > > >> >> >> > is
>>> > > >> >> >> > time to bring this topic up again. I'll try to
>>> > summarize the
>>> > > >> >> >> > (failed)
>>> > > >> >> >> > attempts tried so far:
>>> > > >> >> >> > - the detail=<something> parameter with ad-hoc
>>> > embedding
>>> > > of data.
>>> > > >> >> >> > This
>>> > > >> >> >> > has
>>> > > >> >> >> > been there and removed in 4.0 [3]
>>> > > >> >> >> > - the DoctorREST project - e.g. a proxy above the
>>> > current
>>> > > api. The
>>> > > >> >> >> > idea
>>> > > >> >> >> > was
>>> > > >> >> >> > to create a service which will be independent of
>>> > the engine
>>> > > >> >> >> > itself,
>>> > > >> >> >> > will
>>> > > >> >> >> > locally poll the engine's REST, store all data in
>>> local
>>> > > (mongo)DB
>>> > > >> >> >> > and
>>> > > >> >> >> > provide a rich api with aggregations and
>>> > projections and push
>>> > > >> >> >> > notifications.
>>> > > >> >> >> > This polling of everything to get the data to
>>> > DoctorREST
>>> > > proved to
>>> > > >> >> >> > be
>>> > > >> >> >> > pretty
>>> > > >> >> >> > costy, so also a more invasive approach of pushing
>>> > data from
>>> > > >> >> >> > engine
>>> > > >> >> >> > to
>>> > > >> >> >> > doctor has been discused [5]. None of this two
>>> > approaches
>>> > > have
>>> > > >> >> >> > been
>>> > > >> >> >> > accepted
>>> > > >> >> >> > (too complicated, too invasive).
>>> > > >> >> >> > - writing some custom ad-hoc servlet serving only
>>> a
>>> > > purpose of one
>>> > > >> >> >> > frontend
>>> > > >> >> >> > - this is actually there for the dashboard, but it
>>> > is not a
>>> > > >> >> >> > generic
>>> > > >> >> >> > solution
>>> > > >> >> >> > for the other frontends and we really should not
>>> > develop
>>> > > custom
>>> > > >> >> >> > "APIs"
>>> > > >> >> >> > for
>>> > > >> >> >> > every frontend
>>> > > >> >> >> > - there were some other proposals discussed (some
>>> > 3th party
>>> > > >> >> >> > solutions
>>> > > >> >> >> > etc)
>>> > > >> >> >> > but I think none of them made it even to a PoC
>>> > > >> >> >> >
>>> > > >> >> >> > So, now I would try again and try small to get at
>>> > least some
>>> > > >> >> >> > benefit.
>>> > > >> >> >> > I
>>> > > >> >> >> > see
>>> > > >> >> >> > 2 paths we could try:
>>> > > >> >> >> > 1: embed something which burns us immediatly,
>>> e.g. the
>>> > > /sessions
>>> > > >> >> >> > into
>>> > > >> >> >> > VMs. I
>>> > > >> >> >> > really liked the ;detail=sessions approach, could
>>> > we move
>>> > > it back?
>>> > > >> >> >> > 2: add some tiny service which would just accept a
>>> > list of
>>> > > >> >> >> > queries,
>>> > > >> >> >> > execute
>>> > > >> >> >> > them locally (but using real HTTP requests) and
>>> > return in one
>>> > > >> >> >> > bulk. A
>>> > > >> >> >> > naive
>>> > > >> >> >> > implementation just to give a sense of what I
>>> mean of
>>> > > this would
>>> > > >> >> >> > be a
>>> > > >> >> >> > shell
>>> > > >> >> >> > script getting list of strings like
>>> > > >> >> >> >
>>> > "https://localhost/ovirt-engine/api/vms/123/sessions
>>> > <https://localhost/ovirt-engine/api/vms/123/sessions>
>>> > > <https://localhost/ovirt-engine/api/vms/123/sessions
>>> > <https://localhost/ovirt-engine/api/vms/123/sessions>>" iterate
>>> over
>>> > > >> >> >> > them
>>> > > >> >> >> > and
>>> > > >> >> >> > do a curl request for each, mangle the results
>>> into one
>>> > > string and
>>> > > >> >> >> > return
>>> > > >> >> >> > (credits for this idea to msivak). Easy to
>>> implement,
>>> > > possibility
>>> > > >> >> >> > to
>>> > > >> >> >> > add
>>> > > >> >> >> > also projections later to save some bandwidth.
>>> But the
>>> > > API would
>>> > > >> >> >> > anyway
>>> > > >> >> >> > be
>>> > > >> >> >> > hammered by bunch of queries, only the network
>>> > roundtrip
>>> > > would be
>>> > > >> >> >> > saved.
>>> > > >> >> >> > 3: any other simple approaches?
>>> > > >> >> >> >
>>> > > >> >> >> > I honestly prefer the first approach. It is not
>>> > > beautiful, it is
>>> > > >> >> >> > not
>>> > > >> >> >> > REST-ful, but it is easy to implement, very
>>> > pragmatic and
>>> > > useful.
>>> > > >> >> >> > What do you think?
>>> > > >> >> >> >
>>> > > >> >> >> > Thank you and sorry for the long mail :)
>>> > > >> >> >> > Tomas
>>> > > >> >> >> >
>>> > > >> >> >> > [1]: https://github.com/oVirt/moVirt
>>> > <https://github.com/oVirt/moVirt>
>>> > > <https://github.com/oVirt/moVirt
>>> > <https://github.com/oVirt/moVirt>>
>>> > > >> >> >> > [2]: https://github.com/oVirt/ovirt-web-ui
>>> > <https://github.com/oVirt/ovirt-web-ui>
>>> > > <https://github.com/oVirt/ovirt-web-ui
>>> > <https://github.com/oVirt/ovirt-web-ui>>
>>> > > >> >> >> > [3]: https://gerrit.ovirt.org/#/c/61260
>>> > <https://gerrit.ovirt.org/#/c/61260>
>>> > > <https://gerrit.ovirt.org/#/c/61260
>>> > <https://gerrit.ovirt.org/#/c/61260>>
>>> > > >> >> >> > [4]:
>>> > https://github.com/oVirt/ovirt-web-ui/pull/106/
>>> > <https://github.com/oVirt/ovirt-web-ui/pull/106/>
>>> > > <https://github.com/oVirt/ovirt-web-ui/pull/106/
>>> > <https://github.com/oVirt/ovirt-web-ui/pull/106/>>
>>> > > >> >> >> > [5]: https://gerrit.ovirt.org/#/c/45233/
>>> > <https://gerrit.ovirt.org/#/c/45233/>
>>> > > <https://gerrit.ovirt.org/#/c/45233/
>>> > <https://gerrit.ovirt.org/#/c/45233/>>
>>> > > >> >> >> >
>>> > > >> >> >> >
>>> > > >> >> >> > _______________________________________________
>>> > > >> >> >> > Devel mailing list
>>> > > >> >> >> > Devel at ovirt.org <mailto:Devel at ovirt.org>
>>> > <mailto:Devel at ovirt.org <mailto:Devel at ovirt.org>>
>>> > > >> >> >> > http://lists.ovirt.org/mailman/listinfo/devel
>>> > <http://lists.ovirt.org/mailman/listinfo/devel>
>>> > > <http://lists.ovirt.org/mailman/listinfo/devel
>>> > <http://lists.ovirt.org/mailman/listinfo/devel>>
>>> > > >> >> >> _______________________________________________
>>> > > >> >> >> Devel mailing list
>>> > > >> >> >> Devel at ovirt.org <mailto:Devel at ovirt.org>
>>> > <mailto:Devel at ovirt.org <mailto:Devel at ovirt.org>>
>>> > > >> >> >> http://lists.ovirt.org/mailman/listinfo/devel
>>> > <http://lists.ovirt.org/mailman/listinfo/devel>
>>> > > <http://lists.ovirt.org/mailman/listinfo/devel
>>> > <http://lists.ovirt.org/mailman/listinfo/devel>>
>>> > > >> >> >
>>> > > >> >> >
>>> > > >> >> >
>>> > > >> >> >
>>> > > >> >> > --
>>> > > >> >> > Greg Sheremeta, MBA
>>> > > >> >> > Red Hat, Inc.
>>> > > >> >> > Sr. Software Engineer
>>> > > >> >> > gshereme at redhat.com <mailto:gshereme at redhat.com>
>>> > <mailto:gshereme at redhat.com <mailto:gshereme at redhat.com>>
>>> > > >> >
>>> > > >> >
>>> > > >> _______________________________________________
>>> > > >> Devel mailing list
>>> > > >> Devel at ovirt.org <mailto:Devel at ovirt.org>
>>> > <mailto:Devel at ovirt.org <mailto:Devel at ovirt.org>>
>>> > > >> http://lists.ovirt.org/mailman/listinfo/devel
>>> > <http://lists.ovirt.org/mailman/listinfo/devel>
>>> > > <http://lists.ovirt.org/mailman/listinfo/devel
>>> > <http://lists.ovirt.org/mailman/listinfo/devel>>
>>> > > >
>>> > > >
>>> > >
>>> > >
>>> >
>>> >
>>>
>>>
>>
>> _______________________________________________
>> Devel mailing list
>> Devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>
>
>
> --
> Greg Sheremeta, MBA
> Red Hat, Inc.
> Sr. Software Engineer
> gshereme at redhat.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20170327/c81c1f54/attachment-0001.html>
More information about the Devel
mailing list