On Wed, Nov 30, 2011 at 09:14:16AM +0200, Itamar Heim wrote: > On 11/29/2011 11:36 PM, Adam Litke wrote: >> On Tue, Nov 29, 2011 at 12:54:44PM -0800, Chris Wright wrote: >>> * Adam Litke (agl(a)us.ibm.com) wrote: >>>> On Tue, Nov 29, 2011 at 11:39:08AM -0800, Chris Wright wrote: >>>>> * Adam Litke (agl(a)us.ibm.com) wrote: >>>>>> >>>>>> https://github.com/aglitke/vdsm-rest/ >>>>>> >>>>>> Today I am releasing a proof of concept implementation of a REST API for vdsm. >>>>>> You can find the code on github. My goal is to eventually replace the current >>>>>> xmlrpc interface with a REST API. Once completed, ovirt-engine could switch to >>>>>> this new API. The major advantages to making this change are: 1) VDSM will gain >>>>>> a structured API that conceptually, structurally, and functionally aligns with >>>>>> the ovirt-engine REST API, 2) this new API can be made public, thus providing an >>>>>> entry point for direct virtualization management@the node level. >>>>> >>>>> Adam, this looks like a nice PoC. I didn't see how API versioning is >>>>> handled. Any VDSM developers willing to review this work? >>>> >>>> Thanks for taking a look. I am not handling versioning yet. I think we can add >>>> a version field to the root API object. As for compatibility, we'll just have >>>> to decide on an API backwards-compat support policy. Would this be enough to >>>> handle versioning issues? We shouldn't need anything like capabilities because >>>> the API is discoverable. >>> >>> Right, that seems sensible. >>> >>> Did you find cases where RPC to REST resource mapping was difficult? >> >> I haven't yet fully implemented the current vdsm API but I suspect that certain >> calls (like the ones you mention below) will require some extensions to what I >> have available currently. The main missing piece is probably events and a nice >> polling API. Another big piece of work will be to rebase onto the newly >> redesigned storage model. >> >>> I could see something like migrate() plus migrateStatus() and >>> migrateCancel() needing to add some kind of operational state that to the >>> resource. And something like monitorCommand() which has both a possible >>> side-effect and some freefrom response... >> >> Hopefully monitorCommand will not be too bad, since vdsm should be asking >> libvirt for the VM details when they are needed. Of course we'll need to be >> testing to make sure we aren't keeping state around. Also, I would expect >> monitorCommand to 'taint' the VM in the eyes of the vdsm API (as it does for >> libvirt). >> >>>>> <snip> >>>>>> ovirt-engine wants to subscribe to asynchronous events. REST APIs do not >>>>>> typically support async events and instead rely on polling of resources. I am >>>>>> investigating what options are available for supporting async events via REST. >>>>> >>>>> I think typical is either polling or long polling. If it's a single >>>>> resource, then perhaps long polling would be fine (won't be a large >>>>> number of connections tied up if it's only a single resource). >>>> >>>> Not sure if this is what you are referring to, but I was thinking we could do a >>>> batch-polling mechanism where an API user passes in a list of task UUIDs and/or >>>> event URIs. The server would respond with the status of these resources in one >>>> response. I have some ideas on how we could wire up asynchronous events on the >>>> server side to reduce the amount of actual work that such a batch-polling >>>> operation would require. >>> >>> Oh, I just meant this: >>> >>> Polling (GET /event + 404 loop) >>> Long polling (GET + block ... can chew up a thread connection) >> >> Yep. And we can talk later about building an API for efficient, repeated >> polling. I wonder if the ovirt-engine guys could weigh in as to whether a REST > > cc-ing engine-devel... > >> interface with event polling would be acceptable to them. It is critical that >> we settle on an API that can become _the_ first-class vehicle for interacting >> with vdsm. > > i have two points for consideration around this: > 1. as the api between ovirt-engine and vdsm, I had a preference for the > bus like nature of QMF, as it would allow multiple ovirt-engine to load > balance handling of messages from the queue, and multiple consumers for > some messages (say, history service picking up the stats in parallel to > engine picking them, rather than copying them from engine). I tend to agree, using a bus like QMF between ovirt-engine and vdsm is an inherantly more scalable network architecture, since it avoids the need to have a direct point-to-point connection between the engine and every single node, instead you can build a resilient grid by stategically deploying QPid brokers. As compared to REST, it is also much better at coping with async events, since you can have a push, rather than pull, model which VDSM just puts events onto the bus as they're generated, to be lazily consumed by any remote nodes when desired. NB, I don't mean to imply that there should not *also* be a REST API for the node level. A REST API has the very compelling property that it is trivially accessible from anything with HTTP client support, which is basically everything in existance today. > 2. as node level api, i think a lightweight ovirt-engine managing a > single node and exposing the exact same REST API and behavior of the > multi-node ovirt engine would be easier to cosnume for someone that > wants to interact with a single node same way they would interact with > ovirt-engine. If the REST API is well specified, you wouldn't need to necessarily have a lightweight ovirt-engine deployed at the node level, you could just have VDSM implement the same REST specification natively, as is impl in the engine. Or perhaps provide a some shered code that can be plugged into both VDSM& the enegine to facilitate provision of the same REST interface.

On Wed, Nov 30, 2011 at 03:21:07PM +0200, Itamar Heim wrote: > On 11/30/2011 11:29 AM, Daniel P. Berrange wrote: >> On Wed, Nov 30, 2011 at 09:14:16AM +0200, Itamar Heim wrote: >>> On 11/29/2011 11:36 PM, Adam Litke wrote: >>>> On Tue, Nov 29, 2011 at 12:54:44PM -0800, Chris Wright wrote: >>>>> * Adam Litke (agl(a)us.ibm.com) wrote: >>>>>> On Tue, Nov 29, 2011 at 11:39:08AM -0800, Chris Wright wrote: >>>>>>> * Adam Litke (agl(a)us.ibm.com) wrote: >>>>>>>> >>>>>>>> https://github.com/aglitke/vdsm-rest/ >>>>>>>> >>>>>>>> Today I am releasing a proof of concept implementation of a REST API for vdsm. >>>>>>>> You can find the code on github. My goal is to eventually replace the current >>>>>>>> xmlrpc interface with a REST API. Once completed, ovirt-engine could switch to >>>>>>>> this new API. The major advantages to making this change are: 1) VDSM will gain >>>>>>>> a structured API that conceptually, structurally, and functionally aligns with >>>>>>>> the ovirt-engine REST API, 2) this new API can be made public, thus providing an >>>>>>>> entry point for direct virtualization management@the node level. >>>>>>> >>>>>>> Adam, this looks like a nice PoC. I didn't see how API versioning is >>>>>>> handled. Any VDSM developers willing to review this work? >>>>>> >>>>>> Thanks for taking a look. I am not handling versioning yet. I think we can add >>>>>> a version field to the root API object. As for compatibility, we'll just have >>>>>> to decide on an API backwards-compat support policy. Would this be enough to >>>>>> handle versioning issues? We shouldn't need anything like capabilities because >>>>>> the API is discoverable. >>>>> >>>>> Right, that seems sensible. >>>>> >>>>> Did you find cases where RPC to REST resource mapping was difficult? >>>> >>>> I haven't yet fully implemented the current vdsm API but I suspect that certain >>>> calls (like the ones you mention below) will require some extensions to what I >>>> have available currently. The main missing piece is probably events and a nice >>>> polling API. Another big piece of work will be to rebase onto the newly >>>> redesigned storage model. >>>> >>>>> I could see something like migrate() plus migrateStatus() and >>>>> migrateCancel() needing to add some kind of operational state that to the >>>>> resource. And something like monitorCommand() which has both a possible >>>>> side-effect and some freefrom response... >>>> >>>> Hopefully monitorCommand will not be too bad, since vdsm should be asking >>>> libvirt for the VM details when they are needed. Of course we'll need to be >>>> testing to make sure we aren't keeping state around. Also, I would expect >>>> monitorCommand to 'taint' the VM in the eyes of the vdsm API (as it does for >>>> libvirt). >>>> >>>>>>> <snip> >>>>>>>> ovirt-engine wants to subscribe to asynchronous events. REST APIs do not >>>>>>>> typically support async events and instead rely on polling of resources. I am >>>>>>>> investigating what options are available for supporting async events via REST. >>>>>>> >>>>>>> I think typical is either polling or long polling. If it's a single >>>>>>> resource, then perhaps long polling would be fine (won't be a large >>>>>>> number of connections tied up if it's only a single resource). >>>>>> >>>>>> Not sure if this is what you are referring to, but I was thinking we could do a >>>>>> batch-polling mechanism where an API user passes in a list of task UUIDs and/or >>>>>> event URIs. The server would respond with the status of these resources in one >>>>>> response. I have some ideas on how we could wire up asynchronous events on the >>>>>> server side to reduce the amount of actual work that such a batch-polling >>>>>> operation would require. >>>>> >>>>> Oh, I just meant this: >>>>> >>>>> Polling (GET /event + 404 loop) >>>>> Long polling (GET + block ... can chew up a thread connection) >>>> >>>> Yep. And we can talk later about building an API for efficient, repeated >>>> polling. I wonder if the ovirt-engine guys could weigh in as to whether a REST >>> >>> cc-ing engine-devel... >>> >>>> interface with event polling would be acceptable to them. It is critical that >>>> we settle on an API that can become _the_ first-class vehicle for interacting >>>> with vdsm. >>> >>> i have two points for consideration around this: >>> 1. as the api between ovirt-engine and vdsm, I had a preference for the >>> bus like nature of QMF, as it would allow multiple ovirt-engine to load >>> balance handling of messages from the queue, and multiple consumers for >>> some messages (say, history service picking up the stats in parallel to >>> engine picking them, rather than copying them from engine). >> >> I tend to agree, using a bus like QMF between ovirt-engine and vdsm is >> an inherantly more scalable network architecture, since it avoids the >> need to have a direct point-to-point connection between the engine and >> every single node, instead you can build a resilient grid by stategically >> deploying QPid brokers. >> >> As compared to REST, it is also much better at coping with async events, >> since you can have a push, rather than pull, model which VDSM just puts >> events onto the bus as they're generated, to be lazily consumed by any >> remote nodes when desired. >> >> NB, I don't mean to imply that there should not *also* be a REST API for >> the node level. A REST API has the very compelling property that it is >> trivially accessible from anything with HTTP client support, which is >> basically everything in existance today. >> >>> 2. as node level api, i think a lightweight ovirt-engine managing a >>> single node and exposing the exact same REST API and behavior of the >>> multi-node ovirt engine would be easier to cosnume for someone that >>> wants to interact with a single node same way they would interact with >>> ovirt-engine. >> >> If the REST API is well specified, you wouldn't need to necessarily have >> a lightweight ovirt-engine deployed at the node level, you could just >> have VDSM implement the same REST specification natively, as is impl >> in the engine. Or perhaps provide a some shered code that can be plugged >> into both VDSM& the enegine to facilitate provision of the same REST >> interface. > > Indeed, but an API is not just the transport, it is also behavior > and error code numbering, etc. > trying to develop two different projects trying to do the same thing > sounds like quite the overhead to me. Agreed. > I prefer investing the time in making a lighter version of the > engine for a single node. > then you would also get the web admin and power user portal if you > want them. Could you flesh out some details on the effort required to implement a light-weight engine? I am concerned that this is more complex than it seems. Also, how would we avoid dual maintenance between the full engine and the light one when a new feature/api is added?

My goal is to create a single first-class VDSM API that both ovirt-engine and external applications can use. I am ok with checking out QMF as long as it is distro agnostic and relatively light-weight. In that case, someone (maybe me) could always write a REST interface, but it would merely consume the first-class QMF API. I want to avoid having multiple, parallel APIs to maintain.

On Wed, Nov 30, 2011 at 09:14:16AM +0200, Itamar Heim wrote: > On 11/29/2011 11:36 PM, Adam Litke wrote: >> On Tue, Nov 29, 2011 at 12:54:44PM -0800, Chris Wright wrote: >>> * Adam Litke (agl(a)us.ibm.com) wrote: >>>> On Tue, Nov 29, 2011 at 11:39:08AM -0800, Chris Wright wrote: >>>>> * Adam Litke (agl(a)us.ibm.com) wrote: >>>>>> >>>>>> https://github.com/aglitke/vdsm-rest/ >>>>>> >>>>>> Today I am releasing a proof of concept implementation of a REST API for vdsm. >>>>>> You can find the code on github. My goal is to eventually replace the current >>>>>> xmlrpc interface with a REST API. Once completed, ovirt-engine could switch to >>>>>> this new API. The major advantages to making this change are: 1) VDSM will gain >>>>>> a structured API that conceptually, structurally, and functionally aligns with >>>>>> the ovirt-engine REST API, 2) this new API can be made public, thus providing an >>>>>> entry point for direct virtualization management@the node level. >>>>> >>>>> Adam, this looks like a nice PoC. I didn't see how API versioning is >>>>> handled. Any VDSM developers willing to review this work? >>>> >>>> Thanks for taking a look. I am not handling versioning yet. I think we can add >>>> a version field to the root API object. As for compatibility, we'll just have >>>> to decide on an API backwards-compat support policy. Would this be enough to >>>> handle versioning issues? We shouldn't need anything like capabilities because >>>> the API is discoverable. >>> >>> Right, that seems sensible. >>> >>> Did you find cases where RPC to REST resource mapping was difficult? >> >> I haven't yet fully implemented the current vdsm API but I suspect that certain >> calls (like the ones you mention below) will require some extensions to what I >> have available currently. The main missing piece is probably events and a nice >> polling API. Another big piece of work will be to rebase onto the newly >> redesigned storage model. >> >>> I could see something like migrate() plus migrateStatus() and >>> migrateCancel() needing to add some kind of operational state that to the >>> resource. And something like monitorCommand() which has both a possible >>> side-effect and some freefrom response... >> >> Hopefully monitorCommand will not be too bad, since vdsm should be asking >> libvirt for the VM details when they are needed. Of course we'll need to be >> testing to make sure we aren't keeping state around. Also, I would expect >> monitorCommand to 'taint' the VM in the eyes of the vdsm API (as it does for >> libvirt). >> >>>>> <snip> >>>>>> ovirt-engine wants to subscribe to asynchronous events. REST APIs do not >>>>>> typically support async events and instead rely on polling of resources. I am >>>>>> investigating what options are available for supporting async events via REST. >>>>> >>>>> I think typical is either polling or long polling. If it's a single >>>>> resource, then perhaps long polling would be fine (won't be a large >>>>> number of connections tied up if it's only a single resource). >>>> >>>> Not sure if this is what you are referring to, but I was thinking we could do a >>>> batch-polling mechanism where an API user passes in a list of task UUIDs and/or >>>> event URIs. The server would respond with the status of these resources in one >>>> response. I have some ideas on how we could wire up asynchronous events on the >>>> server side to reduce the amount of actual work that such a batch-polling >>>> operation would require. >>> >>> Oh, I just meant this: >>> >>> Polling (GET /event + 404 loop) >>> Long polling (GET + block ... can chew up a thread connection) >> >> Yep. And we can talk later about building an API for efficient, repeated >> polling. I wonder if the ovirt-engine guys could weigh in as to whether a REST > > cc-ing engine-devel... > >> interface with event polling would be acceptable to them. It is critical that >> we settle on an API that can become _the_ first-class vehicle for interacting >> with vdsm. >> >> Thanks! >> > > > i have two points for consideration around this: > 1. as the api between ovirt-engine and vdsm, I had a preference for > the bus like nature of QMF, as it would allow multiple ovirt-engine > to load balance handling of messages from the queue, and multiple > consumers for some messages (say, history service picking up the > stats in parallel to engine picking them, rather than copying them > from engine). How easy is QMF to consume from a software development perspective? Would it be easy for someone to write a virsh-like tool against a QMF-based vdsm API? Would such a tool be able to run on multiple Linux distributions?

> single node and exposing the exact same REST API and behavior of the > multi-node ovirt engine would be easier to cosnume for someone that > wants to interact with a single node same way they would interact > with ovirt-engine. In principle an ovirt-engine-light sounds great. Especially if we could get the existing GUIs for free. The main concern I have with this is the gargantuan nature of the ovirt-engine and GUI code bases. You want to reserve as many system resources for the guests but it seems that ovirt-engine will consume at least several GB of RAM and one or more CPU cores. How would you lighten up this JBoss stack without rewriting it in a more efficient language? One use case I am interested in is a single-node ovirt instance to control VMs on my laptop. I can spare several hundred MB of RAM at most for the management stack.

On 11/30/2011 04:09 PM, Itamar Heim wrote: >> How easy is QMF to consume from a software development perspective? >> Would it be >> easy for someone to write a virsh-like tool against a QMF-based vdsm >> API? Would >> such a tool be able to run on multiple Linux distributions? > > it is supposed to have a cli console. > cc'ing carl and Ted for more details. I'm not sure how hard it is technically. But for ISV's, I can tell you that almost nobody has experience with it. But, is this intended to be a public API though?

On 11/30/2011 04:09 PM, Itamar Heim wrote: >>How easy is QMF to consume from a software development perspective? >>Would it be >>easy for someone to write a virsh-like tool against a QMF-based vdsm >>API? Would >>such a tool be able to run on multiple Linux distributions? > >it is supposed to have a cli console. >cc'ing carl and Ted for more details. I'm not sure how hard it is technically. But for ISV's, I can tell you that almost nobody has experience with it.

On 11/30/2011 10:52 AM, Adam Litke wrote: >> I'm not sure how hard it is technically. But for ISV's, I can tell >>> you that almost nobody has experience with it. > That is not a good sign. Certainly there must be some sort of standardized and > well understood API transport that we can use. We're not doing anything > particularly novel as far as the API is concerned

There are a bunch of ISV's using it. Doc still lacking in some places but used in massive deployments.

On 11/30/2011 09:09 AM, Itamar Heim wrote: > On 11/30/2011 05:00 PM, Adam Litke wrote: >>> single node and exposing the exact same REST API and behavior of the >>> multi-node ovirt engine would be easier to cosnume for someone that >>> wants to interact with a single node same way they would interact >>> with ovirt-engine. >> >> In principle an ovirt-engine-light sounds great. Especially if we could get the >> existing GUIs for free. The main concern I have with this is the gargantuan >> nature of the ovirt-engine and GUI code bases. You want to reserve as many >> system resources for the guests but it seems that ovirt-engine will consume at >> least several GB of RAM and one or more CPU cores. How would you lighten up >> this JBoss stack without rewriting it in a more efficient language? One use >> case I am interested in is a single-node ovirt instance to control VMs on my >> laptop. I can spare several hundred MB of RAM at most for the management stack. >> > > this is much much lighter and faster with jboss7 (which iirc, takes > 100MB to start, tomcat takes 80MB for comparison). When thinking about something like ovirt-node, my concern is that a jboss7 based ovirt-lite would end up significantly impact the size of the image. I think ovirt-node tends to be around ~100MB today which includes VDSM. A simple REST API (as Adam proposes) wouldn't increase this size at all. Just the openjdk package looks to be around 72MB.