Consumability of vdsm via libvdsm.so

Tue Aug 14 19:46:00 UTC 2012

On Tue, Aug 14, 2012 at 03:26:46PM -0400, Saggi Mizrahi wrote:
> Currently the "strong types" we have are
> * strings
> * uint64 \ int64
> * doubles
> * Booleans
> * array
> * map<string, variant>
> * Null
> 
> On the other hand json has:
> * strings
> * numbers
> * booleans
> * array
> * map<string, variant>
> * null
> 
> We could have all numbers be floats and call it a day but I think that for typed languages the broad distinction we did should still be there.
> 
> Type safety will still be checked by the server side checkers as discussed.
> Checks should be that things are in the correct type and that no mandatory fields are missing from expected objects.

Additionally, fields can only be appended to objects (not removed or changed).
Do we plan to add something to help internal vdsm code construct well-formed
objects which comply to the rules set forth in the API.  Today, objects are
constructed dynamically and in a very ad-hoc manner which makes it difficult to
verify correctness for all possible runtime scenarios.  It seems we would need
to formally construct objects and then "fill in" the values as is appropriate.

> 
> for ZMQ and AMQP I suggest we use the openstack model of listening on 2 topics.
> - vdsm
> - vdsm.hostname

So this is a separate server (analogous to the newly proposed JsonRPC binding)?

> 
> That way you could send a broadcast to all VDSMs or for a specific one. In the future we might messaging specific APIs to have VDSM listen to addtional host groups.
> - vdsm.hostgroup
> 
> This means managers are able to group hosts dynamically.
> These will not be available through the HTTP transport for obvious reasons.
> 
> ----- Original Message -----
> > From: "Adam Litke" <agl at us.ibm.com>
> > To: "Saggi Mizrahi" <smizrahi at redhat.com>
> > Cc: arch at ovirt.org, "Ayal Baron" <abaron at redhat.com>
> > Sent: Tuesday, August 14, 2012 2:57:52 PM
> > Subject: Re: Consumability of vdsm via libvdsm.so
> > 
> > On Tue, Aug 14, 2012 at 02:39:11PM -0400, Saggi Mizrahi wrote:
> > > Well, not by much other then that: * The bindings could sit out of
> > > tree with
> > > less strict support guidelines (TBD).  * We support HTTP but the
> > > bindings
> > > don't use it (unless someone really thinks there is a reason to).
> > >  * You
> > > could. if you so desire. Write against the protocol directly
> > > (optimizations?).
> > > 
> > > We were really close with what we had, the only problems that
> > > people seemed to
> > > have is not committing to a protocol, and forcing everyone to work
> > > with the
> > > bindings.  This means we commit to the protocol and make the
> > > bindings optional
> > > (I don't see why someone would want to not use them, but I know
> > > that
> > > Not-Invented-Here Syndrome is quite common).
> > > 
> > > I feel like it would make everyone happy and it's an evolution of
> > > what we
> > > already started work on.
> > 
> > Ok.  In that case I don't mind committing to a stable protocol as
> > long as we
> > still do bindings for C and Python in the current style because I
> > think we have
> > gotten that part right.  We'll generate native bindings for Java
> > then.  Since
> > JSON is very easy to sloppily extend in incompatible ways (especially
> > for Java/C
> > bindings), how do we plan to enforce strict data types at the
> > protocol level?
> > 
> > Could someone explain to me how the P2P (ZeroMQ/AMQP) stuff fits into
> > this
> > design as well?
> > 
> > > 
> > > ----- Original Message -----
> > > > From: "Adam Litke" <agl at us.ibm.com> To: "Saggi Mizrahi"
> > > > <smizrahi at redhat.com> Cc: arch at ovirt.org, "Ayal Baron"
> > > > <abaron at redhat.com>
> > > > Sent: Tuesday, August 14, 2012 2:11:06 PM Subject: Re:
> > > > Consumability of vdsm
> > > > via libvdsm.so
> > > > 
> > > > On Tue, Aug 14, 2012 at 01:43:01PM -0400, Saggi Mizrahi wrote:
> > > > > 
> > > > > ----- Original Message -----
> > > > > > From: "Adam Litke" <agl at us.ibm.com> To: "Saggi Mizrahi"
> > > > > > <smizrahi at redhat.com> Cc: arch at ovirt.org, "Ayal Baron"
> > > > > > <abaron at redhat.com> Sent: Tuesday, August 14, 2012 12:36:47
> > > > > > PM Subject:
> > > > > > Re: Consumability of vdsm via libvdsm.so
> > > > > > 
> > > > > > On Tue, Aug 14, 2012 at 10:20:34AM -0400, Saggi Mizrahi
> > > > > > wrote:
> > > > > > > I have an Idea, having json rpc for the over HTTP and 2 way
> > > > > > > json RPC
> > > > > > > over openstacks' messaging abstraction layer.
> > > > > > 
> > > > > > So you are proposing to disregard all of the arguments in
> > > > > > favor of a
> > > > > > libvdsm and instead standardize on a protocol.  In that case,
> > > > > > we could
> > > > > > just keep xmlrpc...
> > > > > The difference between xml-rpc and json-rpc is that json-rpc is
> > > > > transport
> > > > > agnostic, less verbose, easier to parse.
> > > > > > We will be back at square one having every application
> > > > > > writing on object
> > > > > > model on top of the transport.  We will also need to reinvent
> > > > > > a model
> > > > > > for ensuring backwards compatibility of the protocol data."
> > > > > As I said, I still think we need to have a schema and supply
> > > > > bindings,
> > > > > GObject for python\C\C++\Vala, and native Java (because, as
> > > > > Yair pointed
> > > > > out, JNI in Java EE is non standard).
> > > > > > 
> > > > > > > This means we have 1 actual code base, the interfaces are
> > > > > > > on the
> > > > > > > transport layer.  And if you choose to use messaging you
> > > > > > > (in the
> > > > > > > future) get 2 way communication which means we can use.
> > > > > > >  Json-RPC
> > > > > > > notification for events. And that we (in the future) might
> > > > > > > fit to an
> > > > > > > openstack cluster.
> > > > > > 
> > > > > > > I still think generating the java bindings is preferable as
> > > > > > > it will
> > > > > > > have to be done and having it generated straight from the
> > > > > > > schema is
> > > > > > > ideal.
> > > > > > 
> > > > > > In that case we will also need to generate python bindings at
> > > > > > a minimum.
> > > > > > In this model, how would REST and AMQP bridges be integrated?
> > > > > No REST, as I said. We forgo the "everything ever can be
> > > > > supported" for
> > > > > json-rpc over different transports. HTTP\AMQP\ZMQ\Simple TCP
> > > > > Socket.
> > > > > Unlike supporting different API schemes supporting transports
> > > > > is fairly
> > > > > simple because the number of methods to implement is small and
> > > > > remains
> > > > > constant and is independent of the actual API.
> > > > > 
> > > > > The big reason to use bindings is to keep us transport
> > > > > agnositc.
> > > > > Something that REST or SOAP or XML-RPC don't give us because
> > > > > they are
> > > > > bound to HTTP.
> > > > > 
> > > > > We could also "extend" on JSON-RPC by being able to switch
> > > > > serializers
> > > > > giving us the ability to support BSON and protobuff as long as
> > > > > the clients
> > > > > adhere to the strong types in the schema.  Unlike XML-RPC that
> > > > > uses
> > > > > advanced XML features, JSON-RPC can really be any serialized
> > > > > object. So
> > > > > technically, if de\serialization becomes an issue we could put
> > > > > high
> > > > > performance serializers as configurable options for very large
> > > > > scale
> > > > > deployments.
> > > > > 
> > > > > JSON-RPC also supports extension methods that can be used to
> > > > > extend the
> > > > > protocol.
> > > > > 
> > > > > As I see it this solves all the issues that were preventing us
> > > > > from
> > > > > committing to a protocoll.  * We don't want to be bound to a
> > > > > single
> > > > > transport. (Transport agnostic) * We don't want to be bound to
> > > > > a single
> > > > > serializer (Solved with optional extension that could be added
> > > > > when we
> > > > > need it) * 2 way communication (if you use anything other then
> > > > > HTTP you
> > > > > have that) * low latency (If you use anything other then HTTP
> > > > > you have
> > > > > that too) * Removal of calls that are actually a group of calls
> > > > > (eg.
> > > > > connectStorageServers vs connectStorageServer) - Even with
> > > > > HTTP, JSON-RPC
> > > > > supports request lists which give the expected semantic.  *
> > > > > Simple set-up.
> > > > > Using ZMQ, HTTP or TCP Socket you can setup VDSM without a
> > > > > broker.  *
> > > > > Don't have a lot of big classes for bindings. Because we bind
> > > > > int the
> > > > > transport level the amount of methods per transport is minimal
> > > > > and
> > > > > constant.  * We provide the bindings. Bindings we'll be
> > > > > available and
> > > > > generated from the schema. So there is no problem with the
> > > > > (everyone needs
> > > > > to write their own bindings).  * Allow bridges. Someone can
> > > > > still write a
> > > > > lightweight REST bridge over the bindings\socket.
> > > > > 
> > > > > Is there something that this doesn't solve?
> > > > 
> > > > Thanks for the additional detail.  How is this proposal different
> > > > from what
> > > > I have submitted to gerrit other than we now need to write an
> > > > additional
> > > > Java bindings generator?
> > > > 
> > > > > > 
> > > > > > > 
> > > > > > > ----- Original Message -----
> > > > > > > > From: "Adam Litke" <agl at us.ibm.com> To: "Ayal Baron"
> > > > > > > > <abaron at redhat.com> Cc: "Saggi Mizrahi"
> > > > > > > > <smizrahi at redhat.com>,
> > > > > > > > arch at ovirt.org Sent: Tuesday, August 14, 2012 10:09:28 AM
> > > > > > > > Subject:
> > > > > > > > Re: Consumability of vdsm via libvdsm.so
> > > > > > > > 
> > > > > > > > On Mon, Aug 13, 2012 at 04:54:18PM -0400, Ayal Baron
> > > > > > > > wrote:
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > ----- Original Message -----
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > ----- Original Message -----
> > > > > > > > > > > From: "Yaniv Kaul" <ykaul at redhat.com> To: "Adam
> > > > > > > > > > > Litke"
> > > > > > > > > > > <agl at us.ibm.com> Cc: arch at ovirt.org Sent: Monday,
> > > > > > > > > > > August 13,
> > > > > > > > > > > 2012 11:49:21 AM Subject: Re: Consumability of vdsm
> > > > > > > > > > > via
> > > > > > > > > > > libvdsm.so
> > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > On 08/13/2012 06:21 PM, Adam Litke wrote:
> > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > Hi all,
> > > > > > > > > > > 
> > > > > > > > > > > We just finished a lively discussion regarding the
> > > > > > > > > > > ongoing
> > > > > > > > > > > effort to stabilize the vdsm API using a C library
> > > > > > > > > > > called
> > > > > > > > > > > libvdsm.  There are many things that need
> > > > > > > > > > > discussion, but I
> > > > > > > > > > > would like to focus this thread on one in
> > > > > > > > > > > particular:
> > > > > > > > > > > 
> > > > > > > > > > > Can ovirt-engine consume libvdsm via JNI?
> > > > > > > > > > > 
> > > > > > > > > > > libvdsm provides a full-featured C interface to the
> > > > > > > > > > > vdsm API
> > > > > > > > > > > using GObject.  Java bindings are provided
> > > > > > > > > > > automatically by
> > > > > > > > > > > jGIR[1].  The library communicates with vdsmd using
> > > > > > > > > > > an
> > > > > > > > > > > internal transport which is not exposed to end
> > > > > > > > > > > users
> > > > > > > > > > > (including ovirt-engine).  I would like to learn
> > > > > > > > > > > from folks
> > > > > > > > > > > with deep Java knowledge if this approach is
> > > > > > > > > > > workable.  What
> > > > > > > > > > > are the technical challenges to integrating in this
> > > > > > > > > > > way?
> > > > > > > > > > > Please save discussion of AMQP and other bindings
> > > > > > > > > > > for other
> > > > > > > > > > > threads.
> > > > > > > > > > > 
> > > > > > > > > > > Thanks!
> > > > > > > > > > > 
> > > > > > > > > > > [1] https://live.gnome.org/JGIR How
> > > > > > > > > > > mature/maintained is JGIR?
> > > > > > > > > > > Last commit seems to be 2010-02-09.  The author is:
> > > > > > > > > > > Alexander
> > > > > > > > > > > Kurtakov <akurtako at redhat.com> His status in our
> > > > > > > > > > > organizational chart: Employee Type: Ex-employee
> > > > > > > > > > It will need some work to get it up to par with the
> > > > > > > > > > rest of the
> > > > > > > > > > gobject generators
> > > > > > > > > 
> > > > > > > > > It's been dead since 2009, that doesn't seem very
> > > > > > > > > promising
> > > > > > > > > (http://www.ohloh.net/p/java-gobject-introspection)
> > > > > > > > > 
> > > > > > > > > It also states: "It also includes a custom GLib/GObject
> > > > > > > > > interface
> > > > > > > > > layer adapted from gstreamer-java." and looking at
> > > > > > > > > gstreamer-java
> > > > > > > > > yields: "An unofficial/alternative set of java bindings
> > > > > > > > > for the
> > > > > > > > > gstreamer multimedia framework"
> > > > > > > > > 
> > > > > > > > > Again, doesn't look like something you want to base
> > > > > > > > > your API on.
> > > > > > > > > Sounds to me like the 'free' java bindings come at a
> > > > > > > > > very high
> > > > > > > > > cost (bringing JGIR up to date and maintaining it).
> > > > > > > > > 
> > > > > > > > > What are the alternatives?
> > > > > > > > 
> > > > > > > > There are two other alternatives (and one bad idea) that
> > > > > > > > come to
> > > > > > > > mind:
> > > > > > > > 
> > > > > > > > 1.) Generate our own Java bindings to libvdsm.so during
> > > > > > > > the vdsm
> > > > > > > > build process.
> > > > > > > > 
> > > > > > > > I prefer this option from an API cleanliness POV because
> > > > > > > > the
> > > > > > > > transport code would only be written once (in
> > > > > > > > libvdsm.so).  This
> > > > > > > > form of generation should be simple because we are just
> > > > > > > > wrapping the
> > > > > > > > library so it can be called via JNI.
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 2.) Generate a native Java "library" that is equivalent
> > > > > > > > to
> > > > > > > > libvdsm.so.
> > > > > > > > 
> > > > > > > > This gets us into the business of libvirt-style bindings
> > > > > > > > generation
> > > > > > > > which I think is a mistake.  It opens the door to us
> > > > > > > > maintaining
> > > > > > > > parallel implementations of "libvdsm"s (one per
> > > > > > > > language).
> > > > > > > > 
> > > > > > > > 
> > > > > > > > -1.) Standardize the transport around JSON-RPC and make
> > > > > > > > that the
> > > > > > > > supported interface.
> > > > > > > > 
> > > > > > > > I am only mentioning it because I am certain someone will
> > > > > > > > bring it
> > > > > > > > up.  I think it's a bad idea and it's off-topic for this
> > > > > > > > particular
> > > > > > > > thread anyway.
> > > > > > > > 
> > > > > > > > -- Adam Litke <agl at us.ibm.com> IBM Linux Technology
> > > > > > > > Center
> > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > > -- Adam Litke <agl at us.ibm.com> IBM Linux Technology Center
> > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > > -- Adam Litke <agl at us.ibm.com> IBM Linux Technology Center
> > > > 
> > > > 
> > > 
> > 
> > --
> > Adam Litke <agl at us.ibm.com>
> > IBM Linux Technology Center
> > 
> > 
> 

-- 
Adam Litke <agl at us.ibm.com>
IBM Linux Technology Center