Consumability of vdsm via libvdsm.so

[Saggi Mizrahi]

----- Original Message -----
> From: "Adam Litke" <agl at us.ibm.com>
> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> Cc: arch at ovirt.org, "Ayal Baron" <abaron at redhat.com>
> Sent: Tuesday, August 14, 2012 3:46:00 PM
> Subject: Re: Consumability of vdsm via libvdsm.so
> 
> On Tue, Aug 14, 2012 at 03:26:46PM -0400, Saggi Mizrahi wrote:
> > Currently the "strong types" we have are
> > * strings
> > * uint64 \ int64
> > * doubles
> > * Booleans
> > * array
> > * map<string, variant>
> > * Null
> > 
> > On the other hand json has:
> > * strings
> > * numbers
> > * booleans
> > * array
> > * map<string, variant>
> > * null
> > 
> > We could have all numbers be floats and call it a day but I think
> > that for typed languages the broad distinction we did should still
> > be there.
> > 
> > Type safety will still be checked by the server side checkers as
> > discussed.
> > Checks should be that things are in the correct type and that no
> > mandatory fields are missing from expected objects.
> 
> Additionally, fields can only be appended to objects (not removed or
> changed).
> Do we plan to add something to help internal vdsm code construct
> well-formed
> objects which comply to the rules set forth in the API.  Today,
> objects are
> constructed dynamically and in a very ad-hoc manner which makes it
> difficult to
> verify correctness for all possible runtime scenarios.  It seems we
> would need
> to formally construct objects and then "fill in" the values as is
> appropriate.
We could check ourselves on the way out as well using some schema generator magic. But I put this as lower priority.
> 
> > 
> > for ZMQ and AMQP I suggest we use the openstack model of listening
> > on 2 topics.
> > - vdsm
> > - vdsm.hostname
> 
> So this is a separate server (analogous to the newly proposed JsonRPC
> binding)?
JsonRPC could still be transmitted over AMQP and ZMQ.
It's the same objects just different transport.
The added benefit is that if such transport is configured VDSM could theoretically send json-rpc notifications (ie. events) to a specified tag.

> 
> > 
> > That way you could send a broadcast to all VDSMs or for a specific
> > one. In the future we might messaging specific APIs to have VDSM
> > listen to addtional host groups.
> > - vdsm.hostgroup
> > 
> > This means managers are able to group hosts dynamically.
> > These will not be available through the HTTP transport for obvious
> > reasons.
> > 
> > ----- Original Message -----
> > > From: "Adam Litke" <agl at us.ibm.com>
> > > To: "Saggi Mizrahi" <smizrahi at redhat.com>
> > > Cc: arch at ovirt.org, "Ayal Baron" <abaron at redhat.com>
> > > Sent: Tuesday, August 14, 2012 2:57:52 PM
> > > Subject: Re: Consumability of vdsm via libvdsm.so
> > > 
> > > On Tue, Aug 14, 2012 at 02:39:11PM -0400, Saggi Mizrahi wrote:
> > > > Well, not by much other then that: * The bindings could sit out
> > > > of
> > > > tree with
> > > > less strict support guidelines (TBD).  * We support HTTP but
> > > > the
> > > > bindings
> > > > don't use it (unless someone really thinks there is a reason
> > > > to).
> > > >  * You
> > > > could. if you so desire. Write against the protocol directly
> > > > (optimizations?).
> > > > 
> > > > We were really close with what we had, the only problems that
> > > > people seemed to
> > > > have is not committing to a protocol, and forcing everyone to
> > > > work
> > > > with the
> > > > bindings.  This means we commit to the protocol and make the
> > > > bindings optional
> > > > (I don't see why someone would want to not use them, but I know
> > > > that
> > > > Not-Invented-Here Syndrome is quite common).
> > > > 
> > > > I feel like it would make everyone happy and it's an evolution
> > > > of
> > > > what we
> > > > already started work on.
> > > 
> > > Ok.  In that case I don't mind committing to a stable protocol as
> > > long as we
> > > still do bindings for C and Python in the current style because I
> > > think we have
> > > gotten that part right.  We'll generate native bindings for Java
> > > then.  Since
> > > JSON is very easy to sloppily extend in incompatible ways
> > > (especially
> > > for Java/C
> > > bindings), how do we plan to enforce strict data types at the
> > > protocol level?
> > > 
> > > Could someone explain to me how the P2P (ZeroMQ/AMQP) stuff fits
> > > into
> > > this
> > > design as well?
> > > 
> > > > 
> > > > ----- Original Message -----
> > > > > From: "Adam Litke" <agl at us.ibm.com> To: "Saggi Mizrahi"
> > > > > <smizrahi at redhat.com> Cc: arch at ovirt.org, "Ayal Baron"
> > > > > <abaron at redhat.com>
> > > > > Sent: Tuesday, August 14, 2012 2:11:06 PM Subject: Re:
> > > > > Consumability of vdsm
> > > > > via libvdsm.so
> > > > > 
> > > > > On Tue, Aug 14, 2012 at 01:43:01PM -0400, Saggi Mizrahi
> > > > > wrote:
> > > > > > 
> > > > > > ----- Original Message -----
> > > > > > > From: "Adam Litke" <agl at us.ibm.com> To: "Saggi Mizrahi"
> > > > > > > <smizrahi at redhat.com> Cc: arch at ovirt.org, "Ayal Baron"
> > > > > > > <abaron at redhat.com> Sent: Tuesday, August 14, 2012
> > > > > > > 12:36:47
> > > > > > > PM Subject:
> > > > > > > Re: Consumability of vdsm via libvdsm.so
> > > > > > > 
> > > > > > > On Tue, Aug 14, 2012 at 10:20:34AM -0400, Saggi Mizrahi
> > > > > > > wrote:
> > > > > > > > I have an Idea, having json rpc for the over HTTP and 2
> > > > > > > > way
> > > > > > > > json RPC
> > > > > > > > over openstacks' messaging abstraction layer.
> > > > > > > 
> > > > > > > So you are proposing to disregard all of the arguments in
> > > > > > > favor of a
> > > > > > > libvdsm and instead standardize on a protocol.  In that
> > > > > > > case,
> > > > > > > we could
> > > > > > > just keep xmlrpc...
> > > > > > The difference between xml-rpc and json-rpc is that
> > > > > > json-rpc is
> > > > > > transport
> > > > > > agnostic, less verbose, easier to parse.
> > > > > > > We will be back at square one having every application
> > > > > > > writing on object
> > > > > > > model on top of the transport.  We will also need to
> > > > > > > reinvent
> > > > > > > a model
> > > > > > > for ensuring backwards compatibility of the protocol
> > > > > > > data."
> > > > > > As I said, I still think we need to have a schema and
> > > > > > supply
> > > > > > bindings,
> > > > > > GObject for python\C\C++\Vala, and native Java (because, as
> > > > > > Yair pointed
> > > > > > out, JNI in Java EE is non standard).
> > > > > > > 
> > > > > > > > This means we have 1 actual code base, the interfaces
> > > > > > > > are
> > > > > > > > on the
> > > > > > > > transport layer.  And if you choose to use messaging
> > > > > > > > you
> > > > > > > > (in the
> > > > > > > > future) get 2 way communication which means we can use.
> > > > > > > >  Json-RPC
> > > > > > > > notification for events. And that we (in the future)
> > > > > > > > might
> > > > > > > > fit to an
> > > > > > > > openstack cluster.
> > > > > > > 
> > > > > > > > I still think generating the java bindings is
> > > > > > > > preferable as
> > > > > > > > it will
> > > > > > > > have to be done and having it generated straight from
> > > > > > > > the
> > > > > > > > schema is
> > > > > > > > ideal.
> > > > > > > 
> > > > > > > In that case we will also need to generate python
> > > > > > > bindings at
> > > > > > > a minimum.
> > > > > > > In this model, how would REST and AMQP bridges be
> > > > > > > integrated?
> > > > > > No REST, as I said. We forgo the "everything ever can be
> > > > > > supported" for
> > > > > > json-rpc over different transports. HTTP\AMQP\ZMQ\Simple
> > > > > > TCP
> > > > > > Socket.
> > > > > > Unlike supporting different API schemes supporting
> > > > > > transports
> > > > > > is fairly
> > > > > > simple because the number of methods to implement is small
> > > > > > and
> > > > > > remains
> > > > > > constant and is independent of the actual API.
> > > > > > 
> > > > > > The big reason to use bindings is to keep us transport
> > > > > > agnositc.
> > > > > > Something that REST or SOAP or XML-RPC don't give us
> > > > > > because
> > > > > > they are
> > > > > > bound to HTTP.
> > > > > > 
> > > > > > We could also "extend" on JSON-RPC by being able to switch
> > > > > > serializers
> > > > > > giving us the ability to support BSON and protobuff as long
> > > > > > as
> > > > > > the clients
> > > > > > adhere to the strong types in the schema.  Unlike XML-RPC
> > > > > > that
> > > > > > uses
> > > > > > advanced XML features, JSON-RPC can really be any
> > > > > > serialized
> > > > > > object. So
> > > > > > technically, if de\serialization becomes an issue we could
> > > > > > put
> > > > > > high
> > > > > > performance serializers as configurable options for very
> > > > > > large
> > > > > > scale
> > > > > > deployments.
> > > > > > 
> > > > > > JSON-RPC also supports extension methods that can be used
> > > > > > to
> > > > > > extend the
> > > > > > protocol.
> > > > > > 
> > > > > > As I see it this solves all the issues that were preventing
> > > > > > us
> > > > > > from
> > > > > > committing to a protocoll.  * We don't want to be bound to
> > > > > > a
> > > > > > single
> > > > > > transport. (Transport agnostic) * We don't want to be bound
> > > > > > to
> > > > > > a single
> > > > > > serializer (Solved with optional extension that could be
> > > > > > added
> > > > > > when we
> > > > > > need it) * 2 way communication (if you use anything other
> > > > > > then
> > > > > > HTTP you
> > > > > > have that) * low latency (If you use anything other then
> > > > > > HTTP
> > > > > > you have
> > > > > > that too) * Removal of calls that are actually a group of
> > > > > > calls
> > > > > > (eg.
> > > > > > connectStorageServers vs connectStorageServer) - Even with
> > > > > > HTTP, JSON-RPC
> > > > > > supports request lists which give the expected semantic.  *
> > > > > > Simple set-up.
> > > > > > Using ZMQ, HTTP or TCP Socket you can setup VDSM without a
> > > > > > broker.  *
> > > > > > Don't have a lot of big classes for bindings. Because we
> > > > > > bind
> > > > > > int the
> > > > > > transport level the amount of methods per transport is
> > > > > > minimal
> > > > > > and
> > > > > > constant.  * We provide the bindings. Bindings we'll be
> > > > > > available and
> > > > > > generated from the schema. So there is no problem with the
> > > > > > (everyone needs
> > > > > > to write their own bindings).  * Allow bridges. Someone can
> > > > > > still write a
> > > > > > lightweight REST bridge over the bindings\socket.
> > > > > > 
> > > > > > Is there something that this doesn't solve?
> > > > > 
> > > > > Thanks for the additional detail.  How is this proposal
> > > > > different
> > > > > from what
> > > > > I have submitted to gerrit other than we now need to write an
> > > > > additional
> > > > > Java bindings generator?
> > > > > 
> > > > > > > 
> > > > > > > > 
> > > > > > > > ----- Original Message -----
> > > > > > > > > From: "Adam Litke" <agl at us.ibm.com> To: "Ayal Baron"
> > > > > > > > > <abaron at redhat.com> Cc: "Saggi Mizrahi"
> > > > > > > > > <smizrahi at redhat.com>,
> > > > > > > > > arch at ovirt.org Sent: Tuesday, August 14, 2012
> > > > > > > > > 10:09:28 AM
> > > > > > > > > Subject:
> > > > > > > > > Re: Consumability of vdsm via libvdsm.so
> > > > > > > > > 
> > > > > > > > > On Mon, Aug 13, 2012 at 04:54:18PM -0400, Ayal Baron
> > > > > > > > > wrote:
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > ----- Original Message -----
> > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > ----- Original Message -----
> > > > > > > > > > > > From: "Yaniv Kaul" <ykaul at redhat.com> To: "Adam
> > > > > > > > > > > > Litke"
> > > > > > > > > > > > <agl at us.ibm.com> Cc: arch at ovirt.org Sent:
> > > > > > > > > > > > Monday,
> > > > > > > > > > > > August 13,
> > > > > > > > > > > > 2012 11:49:21 AM Subject: Re: Consumability of
> > > > > > > > > > > > vdsm
> > > > > > > > > > > > via
> > > > > > > > > > > > libvdsm.so
> > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > On 08/13/2012 06:21 PM, Adam Litke wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > > 
> > > > > > > > > > > > We just finished a lively discussion regarding
> > > > > > > > > > > > the
> > > > > > > > > > > > ongoing
> > > > > > > > > > > > effort to stabilize the vdsm API using a C
> > > > > > > > > > > > library
> > > > > > > > > > > > called
> > > > > > > > > > > > libvdsm.  There are many things that need
> > > > > > > > > > > > discussion, but I
> > > > > > > > > > > > would like to focus this thread on one in
> > > > > > > > > > > > particular:
> > > > > > > > > > > > 
> > > > > > > > > > > > Can ovirt-engine consume libvdsm via JNI?
> > > > > > > > > > > > 
> > > > > > > > > > > > libvdsm provides a full-featured C interface to
> > > > > > > > > > > > the
> > > > > > > > > > > > vdsm API
> > > > > > > > > > > > using GObject.  Java bindings are provided
> > > > > > > > > > > > automatically by
> > > > > > > > > > > > jGIR[1].  The library communicates with vdsmd
> > > > > > > > > > > > using
> > > > > > > > > > > > an
> > > > > > > > > > > > internal transport which is not exposed to end
> > > > > > > > > > > > users
> > > > > > > > > > > > (including ovirt-engine).  I would like to
> > > > > > > > > > > > learn
> > > > > > > > > > > > from folks
> > > > > > > > > > > > with deep Java knowledge if this approach is
> > > > > > > > > > > > workable.  What
> > > > > > > > > > > > are the technical challenges to integrating in
> > > > > > > > > > > > this
> > > > > > > > > > > > way?
> > > > > > > > > > > > Please save discussion of AMQP and other
> > > > > > > > > > > > bindings
> > > > > > > > > > > > for other
> > > > > > > > > > > > threads.
> > > > > > > > > > > > 
> > > > > > > > > > > > Thanks!
> > > > > > > > > > > > 
> > > > > > > > > > > > [1] https://live.gnome.org/JGIR How
> > > > > > > > > > > > mature/maintained is JGIR?
> > > > > > > > > > > > Last commit seems to be 2010-02-09.  The author
> > > > > > > > > > > > is:
> > > > > > > > > > > > Alexander
> > > > > > > > > > > > Kurtakov <akurtako at redhat.com> His status in
> > > > > > > > > > > > our
> > > > > > > > > > > > organizational chart: Employee Type:
> > > > > > > > > > > > Ex-employee
> > > > > > > > > > > It will need some work to get it up to par with
> > > > > > > > > > > the
> > > > > > > > > > > rest of the
> > > > > > > > > > > gobject generators
> > > > > > > > > > 
> > > > > > > > > > It's been dead since 2009, that doesn't seem very
> > > > > > > > > > promising
> > > > > > > > > > (http://www.ohloh.net/p/java-gobject-introspection)
> > > > > > > > > > 
> > > > > > > > > > It also states: "It also includes a custom
> > > > > > > > > > GLib/GObject
> > > > > > > > > > interface
> > > > > > > > > > layer adapted from gstreamer-java." and looking at
> > > > > > > > > > gstreamer-java
> > > > > > > > > > yields: "An unofficial/alternative set of java
> > > > > > > > > > bindings
> > > > > > > > > > for the
> > > > > > > > > > gstreamer multimedia framework"
> > > > > > > > > > 
> > > > > > > > > > Again, doesn't look like something you want to base
> > > > > > > > > > your API on.
> > > > > > > > > > Sounds to me like the 'free' java bindings come at
> > > > > > > > > > a
> > > > > > > > > > very high
> > > > > > > > > > cost (bringing JGIR up to date and maintaining it).
> > > > > > > > > > 
> > > > > > > > > > What are the alternatives?
> > > > > > > > > 
> > > > > > > > > There are two other alternatives (and one bad idea)
> > > > > > > > > that
> > > > > > > > > come to
> > > > > > > > > mind:
> > > > > > > > > 
> > > > > > > > > 1.) Generate our own Java bindings to libvdsm.so
> > > > > > > > > during
> > > > > > > > > the vdsm
> > > > > > > > > build process.
> > > > > > > > > 
> > > > > > > > > I prefer this option from an API cleanliness POV
> > > > > > > > > because
> > > > > > > > > the
> > > > > > > > > transport code would only be written once (in
> > > > > > > > > libvdsm.so).  This
> > > > > > > > > form of generation should be simple because we are
> > > > > > > > > just
> > > > > > > > > wrapping the
> > > > > > > > > library so it can be called via JNI.
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > 2.) Generate a native Java "library" that is
> > > > > > > > > equivalent
> > > > > > > > > to
> > > > > > > > > libvdsm.so.
> > > > > > > > > 
> > > > > > > > > This gets us into the business of libvirt-style
> > > > > > > > > bindings
> > > > > > > > > generation
> > > > > > > > > which I think is a mistake.  It opens the door to us
> > > > > > > > > maintaining
> > > > > > > > > parallel implementations of "libvdsm"s (one per
> > > > > > > > > language).
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > -1.) Standardize the transport around JSON-RPC and
> > > > > > > > > make
> > > > > > > > > that the
> > > > > > > > > supported interface.
> > > > > > > > > 
> > > > > > > > > I am only mentioning it because I am certain someone
> > > > > > > > > will
> > > > > > > > > bring it
> > > > > > > > > up.  I think it's a bad idea and it's off-topic for
> > > > > > > > > this
> > > > > > > > > particular
> > > > > > > > > thread anyway.
> > > > > > > > > 
> > > > > > > > > -- Adam Litke <agl at us.ibm.com> IBM Linux Technology
> > > > > > > > > Center
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > > -- Adam Litke <agl at us.ibm.com> IBM Linux Technology
> > > > > > > Center
> > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > > -- Adam Litke <agl at us.ibm.com> IBM Linux Technology Center
> > > > > 
> > > > > 
> > > > 
> > > 
> > > --
> > > Adam Litke <agl at us.ibm.com>
> > > IBM Linux Technology Center
> > > 
> > > 
> > 
> 
> --
> Adam Litke <agl at us.ibm.com>
> IBM Linux Technology Center
> 
>