On 11/29/2011 11:36 PM, Adam Litke wrote: > On Tue, Nov 29, 2011 at 12:54:44PM -0800, Chris Wright wrote: >> * Adam Litke (agl(a)us.ibm.com) wrote: >>> On Tue, Nov 29, 2011 at 11:39:08AM -0800, Chris Wright wrote: >>>> * Adam Litke (agl(a)us.ibm.com) wrote: >>>>> >>>>> https://github.com/aglitke/vdsm-rest/ >>>>> >>>>> Today I am releasing a proof of concept implementation of a REST API for vdsm. >>>>> You can find the code on github. My goal is to eventually replace the current >>>>> xmlrpc interface with a REST API. Once completed, ovirt-engine could switch to >>>>> this new API. The major advantages to making this change are: 1) VDSM will gain >>>>> a structured API that conceptually, structurally, and functionally aligns with >>>>> the ovirt-engine REST API, 2) this new API can be made public, thus providing an >>>>> entry point for direct virtualization management@the node level. >>>> >>>> Adam, this looks like a nice PoC. I didn't see how API versioning is >>>> handled. Any VDSM developers willing to review this work? >>> >>> Thanks for taking a look. I am not handling versioning yet. I think we can add >>> a version field to the root API object. As for compatibility, we'll just have >>> to decide on an API backwards-compat support policy. Would this be enough to >>> handle versioning issues? We shouldn't need anything like capabilities because >>> the API is discoverable. >> >> Right, that seems sensible. >> >> Did you find cases where RPC to REST resource mapping was difficult? > > I haven't yet fully implemented the current vdsm API but I suspect that certain > calls (like the ones you mention below) will require some extensions to what I > have available currently. The main missing piece is probably events and a nice > polling API. Another big piece of work will be to rebase onto the newly > redesigned storage model. > >> I could see something like migrate() plus migrateStatus() and >> migrateCancel() needing to add some kind of operational state that to the >> resource. And something like monitorCommand() which has both a possible >> side-effect and some freefrom response... > > Hopefully monitorCommand will not be too bad, since vdsm should be asking > libvirt for the VM details when they are needed. Of course we'll need to be > testing to make sure we aren't keeping state around. Also, I would expect > monitorCommand to 'taint' the VM in the eyes of the vdsm API (as it does for > libvirt). > >>>> <snip> >>>>> ovirt-engine wants to subscribe to asynchronous events. REST APIs do not >>>>> typically support async events and instead rely on polling of resources. I am >>>>> investigating what options are available for supporting async events via REST. >>>> >>>> I think typical is either polling or long polling. If it's a single >>>> resource, then perhaps long polling would be fine (won't be a large >>>> number of connections tied up if it's only a single resource). >>> >>> Not sure if this is what you are referring to, but I was thinking we could do a >>> batch-polling mechanism where an API user passes in a list of task UUIDs and/or >>> event URIs. The server would respond with the status of these resources in one >>> response. I have some ideas on how we could wire up asynchronous events on the >>> server side to reduce the amount of actual work that such a batch-polling >>> operation would require. >> >> Oh, I just meant this: >> >> Polling (GET /event + 404 loop) >> Long polling (GET + block ... can chew up a thread connection) > > Yep. And we can talk later about building an API for efficient, repeated > polling. I wonder if the ovirt-engine guys could weigh in as to whether a REST cc-ing engine-devel... > interface with event polling would be acceptable to them. It is critical that > we settle on an API that can become _the_ first-class vehicle for interacting > with vdsm. i have two points for consideration around this: 1. as the api between ovirt-engine and vdsm, I had a preference for the bus like nature of QMF, as it would allow multiple ovirt-engine to load balance handling of messages from the queue, and multiple consumers for some messages (say, history service picking up the stats in parallel to engine picking them, rather than copying them from engine).

2. as node level api, i think a lightweight ovirt-engine managing a single node and exposing the exact same REST API and behavior of the multi-node ovirt engine would be easier to cosnume for someone that wants to interact with a single node same way they would interact with ovirt-engine.

On 11/29/2011 11:36 PM, Adam Litke wrote: >On Tue, Nov 29, 2011 at 12:54:44PM -0800, Chris Wright wrote: >>* Adam Litke (agl(a)us.ibm.com) wrote: >>>On Tue, Nov 29, 2011 at 11:39:08AM -0800, Chris Wright wrote: >>>>* Adam Litke (agl(a)us.ibm.com) wrote: >>>>> >>>>>https://github.com/aglitke/vdsm-rest/ >>>>> >>>>>Today I am releasing a proof of concept implementation of a REST API for vdsm. >>>>>You can find the code on github. My goal is to eventually replace the current >>>>>xmlrpc interface with a REST API. Once completed, ovirt-engine could switch to >>>>>this new API. The major advantages to making this change are: 1) VDSM will gain >>>>>a structured API that conceptually, structurally, and functionally aligns with >>>>>the ovirt-engine REST API, 2) this new API can be made public, thus providing an >>>>>entry point for direct virtualization management@the node level. >>>> >>>>Adam, this looks like a nice PoC. I didn't see how API versioning is >>>>handled. Any VDSM developers willing to review this work? >>> >>>Thanks for taking a look. I am not handling versioning yet. I think we can add >>>a version field to the root API object. As for compatibility, we'll just have >>>to decide on an API backwards-compat support policy. Would this be enough to >>>handle versioning issues? We shouldn't need anything like capabilities because >>>the API is discoverable. >> >>Right, that seems sensible. >> >>Did you find cases where RPC to REST resource mapping was difficult? > >I haven't yet fully implemented the current vdsm API but I suspect that certain >calls (like the ones you mention below) will require some extensions to what I >have available currently. The main missing piece is probably events and a nice >polling API. Another big piece of work will be to rebase onto the newly >redesigned storage model. > >>I could see something like migrate() plus migrateStatus() and >>migrateCancel() needing to add some kind of operational state that to the >>resource. And something like monitorCommand() which has both a possible >>side-effect and some freefrom response... > >Hopefully monitorCommand will not be too bad, since vdsm should be asking >libvirt for the VM details when they are needed. Of course we'll need to be >testing to make sure we aren't keeping state around. Also, I would expect >monitorCommand to 'taint' the VM in the eyes of the vdsm API (as it does for >libvirt). > >>>><snip> >>>>>ovirt-engine wants to subscribe to asynchronous events. REST APIs do not >>>>>typically support async events and instead rely on polling of resources. I am >>>>>investigating what options are available for supporting async events via REST. >>>> >>>>I think typical is either polling or long polling. If it's a single >>>>resource, then perhaps long polling would be fine (won't be a large >>>>number of connections tied up if it's only a single resource). >>> >>>Not sure if this is what you are referring to, but I was thinking we could do a >>>batch-polling mechanism where an API user passes in a list of task UUIDs and/or >>>event URIs. The server would respond with the status of these resources in one >>>response. I have some ideas on how we could wire up asynchronous events on the >>>server side to reduce the amount of actual work that such a batch-polling >>>operation would require. >> >>Oh, I just meant this: >> >>Polling (GET /event + 404 loop) >>Long polling (GET + block ... can chew up a thread connection) > >Yep. And we can talk later about building an API for efficient, repeated >polling. I wonder if the ovirt-engine guys could weigh in as to whether a REST cc-ing engine-devel... >interface with event polling would be acceptable to them. It is critical that >we settle on an API that can become _the_ first-class vehicle for interacting >with vdsm. > >Thanks! > i have two points for consideration around this: 1. as the api between ovirt-engine and vdsm, I had a preference for the bus like nature of QMF, as it would allow multiple ovirt-engine to load balance handling of messages from the queue, and multiple consumers for some messages (say, history service picking up the stats in parallel to engine picking them, rather than copying them from engine).

single node and exposing the exact same REST API and behavior of the multi-node ovirt engine would be easier to cosnume for someone that wants to interact with a single node same way they would interact with ovirt-engine.

On Wed, Nov 30, 2011 at 09:14:16AM +0200, Itamar Heim wrote: > On 11/29/2011 11:36 PM, Adam Litke wrote: > > On Tue, Nov 29, 2011 at 12:54:44PM -0800, Chris Wright wrote: > >> * Adam Litke (agl(a)us.ibm.com) wrote: > >>> On Tue, Nov 29, 2011 at 11:39:08AM -0800, Chris Wright wrote: > >>>> * Adam Litke (agl(a)us.ibm.com) wrote: > >>>>> > >>>>> https://github.com/aglitke/vdsm-rest/ > >>>>> > >>>>> Today I am releasing a proof of concept implementation of a REST API for vdsm. > >>>>> You can find the code on github. My goal is to eventually replace the current > >>>>> xmlrpc interface with a REST API. Once completed, ovirt-engine could switch to > >>>>> this new API. The major advantages to making this change are: 1) VDSM will gain > >>>>> a structured API that conceptually, structurally, and functionally aligns with > >>>>> the ovirt-engine REST API, 2) this new API can be made public, thus providing an > >>>>> entry point for direct virtualization management@the node level. > >>>> > >>>> Adam, this looks like a nice PoC. I didn't see how API versioning is > >>>> handled. Any VDSM developers willing to review this work? > >>> > >>> Thanks for taking a look. I am not handling versioning yet. I think we can add > >>> a version field to the root API object. As for compatibility, we'll just have > >>> to decide on an API backwards-compat support policy. Would this be enough to > >>> handle versioning issues? We shouldn't need anything like capabilities because > >>> the API is discoverable. > >> > >> Right, that seems sensible. > >> > >> Did you find cases where RPC to REST resource mapping was difficult? > > > > I haven't yet fully implemented the current vdsm API but I suspect that certain > > calls (like the ones you mention below) will require some extensions to what I > > have available currently. The main missing piece is probably events and a nice > > polling API. Another big piece of work will be to rebase onto the newly > > redesigned storage model. > > > >> I could see something like migrate() plus migrateStatus() and > >> migrateCancel() needing to add some kind of operational state that to the > >> resource. And something like monitorCommand() which has both a possible > >> side-effect and some freefrom response... > > > > Hopefully monitorCommand will not be too bad, since vdsm should be asking > > libvirt for the VM details when they are needed. Of course we'll need to be > > testing to make sure we aren't keeping state around. Also, I would expect > > monitorCommand to 'taint' the VM in the eyes of the vdsm API (as it does for > > libvirt). > > > >>>> <snip> > >>>>> ovirt-engine wants to subscribe to asynchronous events. REST APIs do not > >>>>> typically support async events and instead rely on polling of resources. I am > >>>>> investigating what options are available for supporting async events via REST. > >>>> > >>>> I think typical is either polling or long polling. If it's a single > >>>> resource, then perhaps long polling would be fine (won't be a large > >>>> number of connections tied up if it's only a single resource). > >>> > >>> Not sure if this is what you are referring to, but I was thinking we could do a > >>> batch-polling mechanism where an API user passes in a list of task UUIDs and/or > >>> event URIs. The server would respond with the status of these resources in one > >>> response. I have some ideas on how we could wire up asynchronous events on the > >>> server side to reduce the amount of actual work that such a batch-polling > >>> operation would require. > >> > >> Oh, I just meant this: > >> > >> Polling (GET /event + 404 loop) > >> Long polling (GET + block ... can chew up a thread connection) > > > > Yep. And we can talk later about building an API for efficient, repeated > > polling. I wonder if the ovirt-engine guys could weigh in as to whether a REST > > cc-ing engine-devel... > > > interface with event polling would be acceptable to them. It is critical that > > we settle on an API that can become _the_ first-class vehicle for interacting > > with vdsm. > > i have two points for consideration around this: > 1. as the api between ovirt-engine and vdsm, I had a preference for the > bus like nature of QMF, as it would allow multiple ovirt-engine to load > balance handling of messages from the queue, and multiple consumers for > some messages (say, history service picking up the stats in parallel to > engine picking them, rather than copying them from engine). I tend to agree, using a bus like QMF between ovirt-engine and vdsm is an inherantly more scalable network architecture, since it avoids the need to have a direct point-to-point connection between the engine and every single node, instead you can build a resilient grid by stategically deploying QPid brokers. As compared to REST, it is also much better at coping with async events, since you can have a push, rather than pull, model which VDSM just puts events onto the bus as they're generated, to be lazily consumed by any remote nodes when desired. NB, I don't mean to imply that there should not *also* be a REST API for the node level. A REST API has the very compelling property that it is trivially accessible from anything with HTTP client support, which is basically everything in existance today.

> 2. as node level api, i think a lightweight ovirt-engine managing a > single node and exposing the exact same REST API and behavior of the > multi-node ovirt engine would be easier to cosnume for someone that > wants to interact with a single node same way they would interact with > ovirt-engine. If the REST API is well specified, you wouldn't need to necessarily have a lightweight ovirt-engine deployed at the node level, you could just have VDSM implement the same REST specification natively, as is impl in the engine. Or perhaps provide a some shered code that can be plugged into both VDSM & the enegine to facilitate provision of the same REST interface.

On Wed, Nov 30, 2011 at 03:21:07PM +0200, Itamar Heim wrote: > On 11/30/2011 11:29 AM, Daniel P. Berrange wrote: >> On Wed, Nov 30, 2011 at 09:14:16AM +0200, Itamar Heim wrote: >>> On 11/29/2011 11:36 PM, Adam Litke wrote: >>>> On Tue, Nov 29, 2011 at 12:54:44PM -0800, Chris Wright wrote: >>>>> * Adam Litke (agl(a)us.ibm.com) wrote: >>>>>> On Tue, Nov 29, 2011 at 11:39:08AM -0800, Chris Wright wrote: >>>>>>> * Adam Litke (agl(a)us.ibm.com) wrote: >>>>>>>> >>>>>>>> https://github.com/aglitke/vdsm-rest/ >>>>>>>> >>>>>>>> Today I am releasing a proof of concept implementation of a REST API for vdsm. >>>>>>>> You can find the code on github. My goal is to eventually replace the current >>>>>>>> xmlrpc interface with a REST API. Once completed, ovirt-engine could switch to >>>>>>>> this new API. The major advantages to making this change are: 1) VDSM will gain >>>>>>>> a structured API that conceptually, structurally, and functionally aligns with >>>>>>>> the ovirt-engine REST API, 2) this new API can be made public, thus providing an >>>>>>>> entry point for direct virtualization management@the node level. >>>>>>> >>>>>>> Adam, this looks like a nice PoC. I didn't see how API versioning is >>>>>>> handled. Any VDSM developers willing to review this work? >>>>>> >>>>>> Thanks for taking a look. I am not handling versioning yet. I think we can add >>>>>> a version field to the root API object. As for compatibility, we'll just have >>>>>> to decide on an API backwards-compat support policy. Would this be enough to >>>>>> handle versioning issues? We shouldn't need anything like capabilities because >>>>>> the API is discoverable. >>>>> >>>>> Right, that seems sensible. >>>>> >>>>> Did you find cases where RPC to REST resource mapping was difficult? >>>> >>>> I haven't yet fully implemented the current vdsm API but I suspect that certain >>>> calls (like the ones you mention below) will require some extensions to what I >>>> have available currently. The main missing piece is probably events and a nice >>>> polling API. Another big piece of work will be to rebase onto the newly >>>> redesigned storage model. >>>> >>>>> I could see something like migrate() plus migrateStatus() and >>>>> migrateCancel() needing to add some kind of operational state that to the >>>>> resource. And something like monitorCommand() which has both a possible >>>>> side-effect and some freefrom response... >>>> >>>> Hopefully monitorCommand will not be too bad, since vdsm should be asking >>>> libvirt for the VM details when they are needed. Of course we'll need to be >>>> testing to make sure we aren't keeping state around. Also, I would expect >>>> monitorCommand to 'taint' the VM in the eyes of the vdsm API (as it does for >>>> libvirt). >>>> >>>>>>> <snip> >>>>>>>> ovirt-engine wants to subscribe to asynchronous events. REST APIs do not >>>>>>>> typically support async events and instead rely on polling of resources. I am >>>>>>>> investigating what options are available for supporting async events via REST. >>>>>>> >>>>>>> I think typical is either polling or long polling. If it's a single >>>>>>> resource, then perhaps long polling would be fine (won't be a large >>>>>>> number of connections tied up if it's only a single resource). >>>>>> >>>>>> Not sure if this is what you are referring to, but I was thinking we could do a >>>>>> batch-polling mechanism where an API user passes in a list of task UUIDs and/or >>>>>> event URIs. The server would respond with the status of these resources in one >>>>>> response. I have some ideas on how we could wire up asynchronous events on the >>>>>> server side to reduce the amount of actual work that such a batch-polling >>>>>> operation would require. >>>>> >>>>> Oh, I just meant this: >>>>> >>>>> Polling (GET /event + 404 loop) >>>>> Long polling (GET + block ... can chew up a thread connection) >>>> >>>> Yep. And we can talk later about building an API for efficient, repeated >>>> polling. I wonder if the ovirt-engine guys could weigh in as to whether a REST >>> >>> cc-ing engine-devel... >>> >>>> interface with event polling would be acceptable to them. It is critical that >>>> we settle on an API that can become _the_ first-class vehicle for interacting >>>> with vdsm. >>> >>> i have two points for consideration around this: >>> 1. as the api between ovirt-engine and vdsm, I had a preference for the >>> bus like nature of QMF, as it would allow multiple ovirt-engine to load >>> balance handling of messages from the queue, and multiple consumers for >>> some messages (say, history service picking up the stats in parallel to >>> engine picking them, rather than copying them from engine). >> >> I tend to agree, using a bus like QMF between ovirt-engine and vdsm is >> an inherantly more scalable network architecture, since it avoids the >> need to have a direct point-to-point connection between the engine and >> every single node, instead you can build a resilient grid by stategically >> deploying QPid brokers. >> >> As compared to REST, it is also much better at coping with async events, >> since you can have a push, rather than pull, model which VDSM just puts >> events onto the bus as they're generated, to be lazily consumed by any >> remote nodes when desired. >> >> NB, I don't mean to imply that there should not *also* be a REST API for >> the node level. A REST API has the very compelling property that it is >> trivially accessible from anything with HTTP client support, which is >> basically everything in existance today. >> >>> 2. as node level api, i think a lightweight ovirt-engine managing a >>> single node and exposing the exact same REST API and behavior of the >>> multi-node ovirt engine would be easier to cosnume for someone that >>> wants to interact with a single node same way they would interact with >>> ovirt-engine. >> >> If the REST API is well specified, you wouldn't need to necessarily have >> a lightweight ovirt-engine deployed at the node level, you could just >> have VDSM implement the same REST specification natively, as is impl >> in the engine. Or perhaps provide a some shered code that can be plugged >> into both VDSM& the enegine to facilitate provision of the same REST >> interface. > > Indeed, but an API is not just the transport, it is also behavior > and error code numbering, etc. > trying to develop two different projects trying to do the same thing > sounds like quite the overhead to me. Agreed. > I prefer investing the time in making a lighter version of the > engine for a single node. > then you would also get the web admin and power user portal if you > want them. Could you flesh out some details on the effort required to implement a light-weight engine? I am concerned that this is more complex than it seems. Also, how would we avoid dual maintenance between the full engine and the light one when a new feature/api is added?

On 11/30/2011 04:09 PM, Itamar Heim wrote: >> How easy is QMF to consume from a software development perspective? >> Would it be >> easy for someone to write a virsh-like tool against a QMF-based vdsm >> API? Would >> such a tool be able to run on multiple Linux distributions? > > it is supposed to have a cli console. > cc'ing carl and Ted for more details. I'm not sure how hard it is technically. But for ISV's, I can tell you that almost nobody has experience with it. But, is this intended to be a public API though?

On 11/30/2011 04:09 PM, Itamar Heim wrote: >>How easy is QMF to consume from a software development perspective? >>Would it be >>easy for someone to write a virsh-like tool against a QMF-based vdsm >>API? Would >>such a tool be able to run on multiple Linux distributions? > >it is supposed to have a cli console. >cc'ing carl and Ted for more details. I'm not sure how hard it is technically. But for ISV's, I can tell you that almost nobody has experience with it. But, is this intended to be a public API though?

On 11/30/2011 09:09 AM, Itamar Heim wrote: > On 11/30/2011 05:00 PM, Adam Litke wrote: >>> single node and exposing the exact same REST API and behavior of the >>> multi-node ovirt engine would be easier to cosnume for someone that >>> wants to interact with a single node same way they would interact >>> with ovirt-engine. >> >> In principle an ovirt-engine-light sounds great. Especially if we could get the >> existing GUIs for free. The main concern I have with this is the gargantuan >> nature of the ovirt-engine and GUI code bases. You want to reserve as many >> system resources for the guests but it seems that ovirt-engine will consume at >> least several GB of RAM and one or more CPU cores. How would you lighten up >> this JBoss stack without rewriting it in a more efficient language? One use >> case I am interested in is a single-node ovirt instance to control VMs on my >> laptop. I can spare several hundred MB of RAM at most for the management stack. >> > > this is much much lighter and faster with jboss7 (which iirc, takes > 100MB to start, tomcat takes 80MB for comparison). When thinking about something like ovirt-node, my concern is that a jboss7 based ovirt-lite would end up significantly impact the size of the image. I think ovirt-node tends to be around ~100MB today which includes VDSM. A simple REST API (as Adam proposes) wouldn't increase this size at all. Just the openjdk package looks to be around 72MB.

My goal is to create a single first-class VDSM API that both ovirt-engine and external applications can use. I am ok with checking out QMF as long as it is distro agnostic and relatively light-weight. In that case, someone (maybe me) could always write a REST interface, but it would merely consume the first-class QMF API. I want to avoid having multiple, parallel APIs to maintain.

On 11/30/2011 10:52 AM, Adam Litke wrote: >> I'm not sure how hard it is technically. But for ISV's, I can tell >>> you that almost nobody has experience with it. > That is not a good sign. Certainly there must be some sort of standardized and > well understood API transport that we can use. We're not doing anything > particularly novel as far as the API is concerned

There are a bunch of ISV's using it. Doc still lacking in some places but used in massive deployments.