[ovirt-devel] [ATN] Introduction of RESTAPI metamodel

Roman Mohr rmohr at redhat.com
Tue Oct 27 11:55:49 UTC 2015


On Tue, Oct 27, 2015 at 12:16 PM, Juan Hernández <jhernand at redhat.com>
wrote:

> On 10/27/2015 11:28 AM, Roman Mohr wrote:
> >
> >
> > On Tue, Oct 27, 2015 at 10:47 AM, Juan Hernández <jhernand at redhat.com
> > <mailto:jhernand at redhat.com>> wrote:
> >
> >     On 10/27/2015 10:16 AM, Roman Mohr wrote:
> >     >
> >     >
> >     > On Mon, Oct 26, 2015 at 5:32 PM, Juan Hernández <
> jhernand at redhat.com <mailto:jhernand at redhat.com>
> >     > <mailto:jhernand at redhat.com <mailto:jhernand at redhat.com>>> wrote:
> >     >
> >     >     On 10/26/2015 04:56 PM, Roman Mohr wrote:
> >     >     > Hi Juan,
> >     >     >
> >     >     > The way to specify the contract look pretty clean and nice.
> >     >     > I would love to read a few words about the big picture. What
> >     is the
> >     >     > final scenario?
> >     >     >
> >     >
> >     >     The motivation for this change is that currently we don't have
> >     a central
> >     >     place where the RESTAPI is specified, rather we have several
> >     different
> >     >     places, using several different technologies:
> >     >
> >     >     * XML schema for the data model.
> >     >     * JAX-RS for part of the operational model (without the
> >     parameters).
> >     >     * rsdl_metadata.yaml for the parameters of the operational
> model.
> >     >
> >     >     This makes it difficult to infer information about the model.
> For
> >     >     example, the generators of the SDKs have to download the XML
> >     schema, and
> >     >     the RSDL (which is generated from the JAX-RS interfaces using
> >     reflection
> >     >     and combining it with the information from the
> >     rsdl_metadata.yaml file)
> >     >     and then they have to do their own computations to extract
> >     what they
> >     >     need.
> >     >
> >     >     Same happens with the CLI: it has to extract the information
> >     it needs
> >     >     from the Python code generated for the Python SDK, yet another
> >     level of
> >     >     indirection.
> >     >
> >     >
> >     > You are right, that definitely needs to be cleaned up. I just want
> to
> >     > discuss a few points below with you.
> >     >
> >     >
> >     >
> >     >     We are also lacking a comprehensive reference documentation of
> the
> >     >     RESTAPI. What we currently have has been written by hand, and
> >     gets out
> >     >     of sync very quickly, and we don't even notice.
> >     >
> >     >
> >     > Did you also consider swagger? It is made for exactly that purpose.
> >     > I created a demo in [1] which uses resteasy, weld,
> hibernate-validator
> >     > and swagger to demonstrate how to do DRY with jaxrs.
> >     > Would be great to hear you thoughts on that.
> >     >
> >     > And there is the great swagger-ui [8] to display the documentation
> >     in a
> >     > more human readable way.
> >     >
> >
> >     Yes, I considered Swagger, and rejected it because it is JSON
> centric,
> >     and I think JSON isn't as good as Java to represent the contracts of
> our
> >     RESTAPI.
> >
> >
> > You just write plain jax-rs, swagger just creates a description out of
> > it. So  the source defining the contract is pure java (jax-rs with some
> > swagger annotations for description, etc.).
> > Or am I missing the point here?
> >
>
> If I understand correctly the Swagger core is a JSON (or YAML)
> specification of the API. From that you can generate JAX-RS annotated
> code, not the other way around. So the specification document that you
> write is a JSON document.
>

You are right, my terminology here was not clear. Swagger is just a
specification. Swagger-core and swagger-jaxrs are the ones which can create
the documnetation out of JAX-RS resources.


> Alternatively, you can use the Swagger annotations to decorate your
> implementation, both the entity classes and the JAX-RS resource
> implementations, and then extract the model from that. But this is
> putting the implementation before the specification. That is where we
> are today, and it causes multiple problems. I think it is better to have
> the specification and the implementation separate. Swagger does that
> well when using JSON directly, our metamodel also does it well, but
> using a better language.
>


Isn't our problem that we have everything scattered arount the place and
not that we are using JAX-RS? I don't think that this has anything to do
with specification before implementation or implementation before
specification.


> >
> >
> >     In addition we need to do these changes in a smooth way, without
> causing
> >     big changes in the middle. For example, in the first step we need to
> >     preserve the JAX-RS interfaces as they are today, to avoid massive
> >     changes to all the resource implementations. This could be done with
> >
> >     Swagger, but would require custom code generators. With less effort
> we
> >     can do our own.
> >
> >
> > This is of course generally a difficult task. But I do not know why it
> > would be more difficult to write a custom swagger reader (if we even
> > have to, it can read the interfaces as well) .
> > They are pretty streight forward. Just look at [9], this contains the
> > wole jax-rs specific code to generate the swagger documentation.
> >
> > But yes, I don't know every detail here of the engine and can't clearly
> > say that integrating that would just streight forward (my feeling tells
> > me that it would not be too hard). I am just under the impression that
> > we would benefit from that. Just reduces custom magic to a minimum.
> >
>
> Using something like Swagger would be certainly possible, and not that
> hard, but it requires an effort. For example, say that we decide to use
> the Swagger annotations. Then we will need to add these annotations to
> all our JAX-RS resource implementations. That is a non trivial effort.
> We would need to add the annotations to the entities as well. But wait,
> we don't have such entities, only XML schema. So we would need a reader
> that parses XML schema, and creating it requires effort.


You can just create the entities/daos once like we do now on every build
and annotate them once and drop the whole xml.


> Where do we put
> the documentation then? Part in the JAX-RS interfaces, part in the XML
> schema.


I don't understand that. There is only one documentation and implementation
source, that is the JAX-RS resource and the entity/dao accepted by the
endpoint.
The XML schema can just be removed. The documentation in swagger format for
other tools like swagger-codegen and swagger-ci can be made available
through swagger-maven-plugin or a servlet in the engine which generates the
swagger json on the fly (like I did in [1])


> We are already there, and we ended up with no documentation at
> all. In my view the sum of these efforts is higher than doing our own
> metamodel.
>


>
>
>
> >
> >     Swagger UI is certainly great. I did test it and it is really good.
> We
> >     may be able to copy some concepts.
> >
> >     >
> >     >
> >     >     To solve these issues I intend to have the specification of
> >     the RESTAPI
> >     >     only in one place, and using only one technology. I decided to
> >     use Java
> >     >     interfaces for that. Note however that they are just the
> >     support for the
> >     >     information, like paper is the support for ink. I decided to
> >     use Java
> >     >     because it is easy to create, modify and re-factor using tools
> >     familiar
> >     >     to most of us.
> >     >
> >     >     These source of these interfaces is analysed (using QDox,
> >     currently) and
> >     >     a "model" of the RESTAPI is generated in memory. This model is
> >     >     independent of the supporting Java source, and easy to
> >     consume. For
> >     >     example, imagine that you want to list all the types available
> >     in the
> >     >     model and for each one display its documentation:
> >     >
> >     >       Model model = ...;
> >     >       for (Type type : model.getTypes()) {
> >     >         Name name = type.getName();
> >     >         String doc = type.getDoc();
> >     >         System.out.println(name + ": " + doc);
> >     >       }
> >     >
> >     >     Something like this, but more elaborate, will be part of a web
> >     >     application that provides comprehensive reference
> documentation,
> >     >     assuming that we dedicate the time to write documentation
> >     comments in
> >     >     the specification.
> >     >
> >     >     I intend to use this model also to do simplify the generators
> >     of the
> >     >     SDKs and the CLI.
> >     >
> >     >     In addition these are some of the things that I would like to
> >     change in
> >     >     the near future (for 4.0):
> >     >
> >     >     * Move the specification of the parameters of operations out
> >     of the
> >     >     rsdl_metadata.yaml file and into the model. For example:
> >     >
> >     >       @Service
> >     >       public VmService {
> >     >         /**
> >     >          * The operation to add a virtual machine.
> >     >          */
> >     >         interface Add {
> >     >           /**
> >     >            * The representation of the virtual machine is received
> >     >            * as parameter, and the representation of the created
> >     >            * virtual machine is returned as result.
> >     >            */
> >     >            @In @Out Vm vm();
> >     >
> >     >            /**
> >     >             * In the future, we will be able to specify other
> >     >             * parameters here.
> >     >             */
> >     >            @In Boolean force();
> >     >
> >     >            /**
> >     >             * Even with default values.
> >     >             */
> >     >            @In default Boolean force() { return true; }
> >     >
> >     >            /**
> >     >             * And we will be able to specify constraints, which
> >     >             * will replace the rsdl_metadata.yaml file.
> >     >             */
> >     >            @Constraint
> >     >            default boolean vmNameMustNotBeNull() {
> >     >              return vm().name() != null;
> >     >            }
> >     >          }
> >     >       }
> >     >
> >     >     * Enforce the constraints automatically. If the constraints
> >     are in the
> >     >     model, then we can just check them and reject requests before
> >     delivering
> >     >     them to the application. Currently we do this manually (and
> often
> >     >     forget) with calls to "validate(...)" methods.
> >     >
> >     >
> >     >
> >     > Did you consider just annotating the DTOs with JSR-303 annotations
> and
> >     > integrate a validator with jax-rs?
> >     > See [2] for an example.
> >     >
> >
> >     This is a great way to implement a system, but the goal here isn't to
> >     implement it, rather to specify it. Using annotations in this way
> won't
> >     help the generators of the SDKs, for example, to figure out what
> >     parameters are required, mandatory, etc.
> >
> >
> > Swagger understands them. From my example project, swagger created that
> >
> >      description:
> >         type: "string"
> >         minLength: 10
> >         maxLength: 100
> >
> > out of
> >
> >      @Size(min=10, max=100) # jsr-303
> >      private String description;
> >
> > and so does swagger-codegen which can generate clients in java, python,
> ...
> >
>
> This is extracting the specification from the implementation, which
> isn't correct in my opinion, it should be the opposite. Not saying that
> this makes Swagger bad, it is nice that it has this capability, but I
> think we can do it better.
>

In my opition this is the main advantage of that. It is DRY while still
having full control of the implementation.


> >     >
> >     >
> >     >     * Generate the Java classes directly from the model. Instead
> of Model ->
> >     >     XML Schema -> Java, we can do Model -> Java. This will allow
> us to solve
> >     >     some of the XJC compiler limitations, like the horrible way we
> handle
> >     >     arrays today.
> >     >
> >     >
> >     > Swagger [3] is a rest documentation specification. There is also a
> maven
> >     > plugin [4] and you can create clients for example with [5].
> >     >
> >     >
> >     >
> >     >     * Replace JAX-RS with a simpler infrastructure that supports
> better
> >     >     streaming and CDI injection.
> >     >
> >     >
> >     >
> >     > With resteasy-cdi you have pretty good injection support for
> resteasy.
> >     > Run the demo in [1] to see it in action and look at the file at
> [6].
> >     >
> >
> >     Resteasy-CDI isn't standard, it only works with Resteasy. If we rely
> on
> >     it then we re tied to Resteasy for ever.
> >
> >
> > Even jersey has support for that (I think it is called jeryse-gf-cdi),
> > but why would we want switch? I don't think that jboss will drop
> > resteasy and it also works fine outside of full blown containers. I
> > don't think that this is an argument.
> >
>
> Well, nobody thought that JBoss would drop Tomcat, and they did. Nobody
> thought that Resteasy would change the SPI from 2.x to 3.x, and they did.\
>

It will be there for the next year or another library which offers the same
thing. Having injections in Jax-rs resources is important for spring,
jboss, glassfish and others there will always be ways to do that. I don't
know why we should create our own 'custom standard' just because another
standard does not include dependency injection when we can just extend it
so easily.


> We want (well, I want) to get out of JAX-RS because it doesn't support
> well CDI and streaming.
>

Why should streaming resources be in the rest interface?

>
> >
> >     >
> >     >
> >     >     * Add support for multiple versions of the API, using the
> "Version"
> >     >     header, and generating different Java classes for entities and
> services.
> >     >     For example, if we have versions 4 and 5 of the model as
> separate
> >     >     artifacts, then we can generate "V4Vm" and "V5Vm" entity
> classes, and
> >     >     "V4VmService" and "V5VmService" service classes. These can be
> used
> >     >     simultaneously in the server, so we can have in the same engine
> >     >     implementations for multiple versions.
> >     >
> >     >
> >     > There are also many ways to do that. Here [7] is a pretty clean
> way to
> >     > do it with jax-rs and you will have everything related in one
> resource.
> >     >
> >
> >     Yes, there are many ways. In my opinion it is better to use the HTTP
> >     "Version" header, and to forward requests to different resource
> >     implementations without requiring different URLs or different
> >     content types.
> >
> > Have no strong opinion there, just seemed to be a good choice regarding
> > to versioning limitations in jax-rs and our use of jax-rs subresources.
> >
> >
> >     >
> >     >
> >     >     The final picture isn't completely defined yet.
> >     >
> >     >     Regards,
> >     >     Juan Hernandez
> >     >
> >     >     > On Mon, Oct 26, 2015 at 4:03 PM, Juan Hernández <
> jhernand at redhat.com <mailto:jhernand at redhat.com>
> >     <mailto:jhernand at redhat.com <mailto:jhernand at redhat.com>>
> >     >     > <mailto:jhernand at redhat.com <mailto:jhernand at redhat.com>
> >     <mailto:jhernand at redhat.com <mailto:jhernand at redhat.com>>>> wrote:
> >     >     >
> >     >     >     Hello,
> >     >     >
> >     >     >     I will soon merge the following patches that introduce a
> new
> >     >     way to
> >     >     >     specify the contracts of the RESTAPI:
> >     >     >
> >     >     >       restapi: Introduce metamodel
> >     >     >       https://gerrit.ovirt.org/45852
> >     >     >
> >     >     >       restapi: Use metamodel
> >     >     >       https://gerrit.ovirt.org/46478
> >     >     >
> >     >     >       restapi: Generate JAX-RS interfaces from model
> >     >     >       https://gerrit.ovirt.org/47337
> >     >     >
> >     >     >
> >     >
> >     >     > Looks pretty much like we are replacing one way of
> >     annotating things
> >     >     > with another way of specifying things.
> >     >     > Could you elaborate what the benefit of that way of
> >     description is?
> >     >     >
> >     >     > How would I customize endpoints with e.g. @Gzip annotations?
> >     Would
> >     >     I at
> >     >     > the end still have my JAX-RS annotates resource classes?
> >     >     >
> >     >     >
> >     >     >     These patches introduce a new "metamodel" concept, and
> move
> >     >     the current
> >     >     >     specification of the RESTAPI based on XML schema and
> JAX-RS
> >     >     interfaces
> >     >     >     to a new "model" built on the new metamodel.
> >     >     >
> >     >     >
> >     >     >     What does this mean for you in practical terms?
> >     Currently when
> >     >     you want
> >     >     >     to introduce or modify one of the data types used by the
> >     >     RESTAPI you
> >     >     >     start by modifying the XML schema. Once the patches are
> >     merged
> >     >     the XML
> >     >     >     schema will never be touched, as it will be automatically
> >     >     generated from
> >     >     >     the "model". For example, imagine that you need to add a
> new
> >     >     "color"
> >     >     >     attribute to the "VM" entity. To do so with the new
> >     model you
> >     >     will have
> >     >     >     to modify the following file, which is the specification
> of
> >     >     the "Vm"
> >     >     >     entity, written as a Java interface:
> >     >     >
> >     >     >
> >     >     >
> >     >
> >
> https://gerrit.ovirt.org/#/c/46478/16/backend/manager/modules/restapi/model/src/main/java/types/Vm.java
> >     >     >
> >     >     >     In that interface you will have to add a line like this:
> >     >     >
> >     >     >       String color();
> >     >     >
> >     >     >     Note that this Java interface is just the specification
> >     of the
> >     >     entity,
> >     >     >     it won't be used at all during runtime. Instead of that
> the
> >     >     XML schema
> >     >     >     will be generated from it, and then Java will be
> generated
> >     >     from the XML
> >     >     >     schema, as we do today (this will change in the future,
> but
> >     >     not yet).
> >     >     >
> >     >     >     Same for the services. If you want to add a new "paint"
> >     action
> >     >     to the
> >     >     >     "Vm" resource then you won't modify the JAX-RS
> interfaces,
> >     >     instead of
> >     >     >     that you will modify the following file, which is the
> >     >     specification of
> >     >     >     the "Vm" service, written as a Java interface:
> >     >     >
> >     >     >
> >     >     >
> >     >
> >
> https://gerrit.ovirt.org/#/c/47337/6/backend/manager/modules/restapi/model/src/main/java/services/VmService.java
> >     >     >
> >     >     >     In that interface you will need to add a sub-interface
> >     >     representing the
> >     >     >     action:
> >     >     >
> >     >     >       interface Paint {
> >     >     >       }
> >     >     >
> >     >     >     The JAX-RS interface will be generated from that.
> >     Currently these
> >     >     >     sub-interfaces are empty. In the future they will
> >     contain the
> >     >     >     specifications of the parameters (currently in the
> >     >     rsdl_metadata.yml
> >     >     >     file).
> >     >     >
> >     >     >
> >     >     >
> >     >     >     These changes will currently affect only the
> >     specification of the
> >     >     >     RESTAPI, not the implementation, so in in the
> >     >     "Backend*Resource" classes
> >     >     >     things won't change yet.
> >     >     >
> >     >     >
> >     >     > Currently I do not really understand where we are going
> >     here. Are we
> >     >     > trying to get rid of rdsl?
> >     >     >
> >     >     > So basically two questions:
> >     >     >
> >     >     > 1) What is the final goal?
> >     >     > 2) What speaks agains using Hibernate validator on Daos in
> >     combination
> >     >     > with JAX-RS annotated resources (and just removing all
> >     interfaces, as
> >     >     > far as I can see we only have one implementation per
> >     endpoint) and
> >     >     > creating all schemas and clients through SWAGGER tooling?
> >     >     >
> >     >     >
> >     >     >     If you have doubts, please let me know.
> >     >     >
> >     >     >     Regards,
> >     >     >     Juan Hernandez
> >     >     >
> >     >     >     --
> >     >     >     Dirección Comercial: C/Jose Bardasano Baos, 9, Edif.
> >     Gorbea 3,
> >     >     planta
> >     >     >     3ºD, 28016 Madrid, Spain
> >     >     >     Inscrita en el Reg. Mercantil de Madrid – C.I.F.
> B82657941 -
> >     >     Red Hat
> >     >     >     S.L.
> >     >     >     _______________________________________________
> >     >     >     Devel mailing list
> >     >     >     Devel at ovirt.org <mailto:Devel at ovirt.org>
> >     <mailto:Devel at ovirt.org <mailto:Devel at ovirt.org>>
> >     >     <mailto:Devel at ovirt.org <mailto:Devel at ovirt.org>
> >     <mailto:Devel at ovirt.org <mailto:Devel at ovirt.org>>>
> >     >     >     http://lists.ovirt.org/mailman/listinfo/devel
> >     >     >
> >     >     >
> >     >     > Thanks,
> >     >     >
> >     >     > Roman
> >     >
> >     >
> >     >     --
> >     >     Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3,
> >     planta
> >     >     3ºD, 28016 Madrid, Spain
> >     >     Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 -
> >     Red Hat
> >     >     S.L.
> >     >
> >     >
> >     >
> >     > I don't know if it is the right thing to do to invent something new
> >     > here. I personally would prefer to thread a path which is very
> >     common on
> >     > the java community.
> >     > I would love follow the DRY principle regarding to the stack and
> the
> >     > code and would just use the great community projects there.
> >     >
> >     > It would also completely eliminate any custom magic. The JAX-RS
> >     and CDI
> >     > magic is pretty standard and easy to understand.
> >     > From my perspective, real JAX-RS resoures have the advantage of
> >     >
> >     >  * being very easy to understand (there is magic, but the
> >     connection to
> >     > the real endpoint is pretty clear)
> >     >  * being easy to customize suff, like adding @GZip to an annotation
> >     >  * describing pretty clearly the connection between the generated
> rest
> >     > interface and the internal services
> >     >
> >     > Finally writing hand crafted tests is also much easier.
> >     >
> >     > What are your thoughts about that?
> >     >
> >     > Best Regards,
> >     > Roman
> >     >
> >     >
> >     > [1] https://github.com/rmohr/jetty-maven-cdi-demo
> >     > [2]
> >     >
> >
> https://github.com/rmohr/jetty-maven-cdi-demo/blob/master/src/main/java/rmohr/examples/cdi/MyDto.java
> >     > [3] http://swagger.io/
> >     > [4] https://github.com/kongchen/swagger-maven-plugin
> >     > [5] https://github.com/swagger-api/swagger-codegen
> >     > [6]
> >     >
> >
> https://github.com/rmohr/jetty-maven-cdi-demo/blob/master/src/main/java/rmohr/examples/cdi/RestSubResource.java
> >     > [7]
> >     >
> >
> http://maxenglander.com/2013/04/23/basic-restful-api-versioning-in-jersey.html
> >     > [8] https://github.com/swagger-api/swagger-ui
> >     >
> >
> > [9]
> >
> https://github.com/swagger-api/swagger-core/blob/master/modules/swagger-jaxrs/src/main/java/io/swagger/jaxrs/Reader.java
> >
>
> --
> Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta
> 3ºD, 28016 Madrid, Spain
> Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20151027/51d85cff/attachment-0001.html>


More information about the Devel mailing list