On 10/27/2015 10:16 AM, Roman Mohr wrote:

On Mon, Oct 26, 2015 at 5:32 PM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com>> wrote:

On 10/26/2015 04:56 PM, Roman Mohr wrote: > Hi Juan, > > The way to specify the contract look pretty clean and nice. > I would love to read a few words about the big picture. What is the > final scenario? >

The motivation for this change is that currently we don't have a central place where the RESTAPI is specified, rather we have several different places, using several different technologies:

* XML schema for the data model. * JAX-RS for part of the operational model (without the parameters). * rsdl_metadata.yaml for the parameters of the operational model.

This makes it difficult to infer information about the model. For example, the generators of the SDKs have to download the XML schema, and the RSDL (which is generated from the JAX-RS interfaces using reflection and combining it with the information from the rsdl_metadata.yaml file) and then they have to do their own computations to extract what they need.

Same happens with the CLI: it has to extract the information it needs from the Python code generated for the Python SDK, yet another level of indirection.

You are right, that definitely needs to be cleaned up. I just want to discuss a few points below with you.

We are also lacking a comprehensive reference documentation of the RESTAPI. What we currently have has been written by hand, and gets out of sync very quickly, and we don't even notice.

Did you also consider swagger? It is made for exactly that purpose. I created a demo in [1] which uses resteasy, weld, hibernate-validator and swagger to demonstrate how to do DRY with jaxrs. Would be great to hear you thoughts on that.

And there is the great swagger-ui [8] to display the documentation in a more human readable way.

Yes, I considered Swagger, and rejected it because it is JSON centric, and I think JSON isn't as good as Java to represent the contracts of our RESTAPI. In addition we need to do these changes in a smooth way, without causing big changes in the middle. For example, in the first step we need to preserve the JAX-RS interfaces as they are today, to avoid massive changes to all the resource implementations. This could be done with Swagger, but would require custom code generators. With less effort we can do our own. Swagger UI is certainly great. I did test it and it is really good. We may be able to copy some concepts.

To solve these issues I intend to have the specification of the RESTAPI only in one place, and using only one technology. I decided to use Java interfaces for that. Note however that they are just the support for the information, like paper is the support for ink. I decided to use Java because it is easy to create, modify and re-factor using tools familiar to most of us.

These source of these interfaces is analysed (using QDox, currently) and a "model" of the RESTAPI is generated in memory. This model is independent of the supporting Java source, and easy to consume. For example, imagine that you want to list all the types available in the model and for each one display its documentation:

Model model = ...; for (Type type : model.getTypes()) { Name name = type.getName(); String doc = type.getDoc(); System.out.println(name + ": " + doc); }

Something like this, but more elaborate, will be part of a web application that provides comprehensive reference documentation, assuming that we dedicate the time to write documentation comments in the specification.

I intend to use this model also to do simplify the generators of the SDKs and the CLI.

In addition these are some of the things that I would like to change in the near future (for 4.0):

* Move the specification of the parameters of operations out of the rsdl_metadata.yaml file and into the model. For example:

@Service public VmService { /** * The operation to add a virtual machine. */ interface Add { /** * The representation of the virtual machine is received * as parameter, and the representation of the created * virtual machine is returned as result. */ @In @Out Vm vm();

/** * In the future, we will be able to specify other * parameters here. */ @In Boolean force();

/** * Even with default values. */ @In default Boolean force() { return true; }

/** * And we will be able to specify constraints, which * will replace the rsdl_metadata.yaml file. */ @Constraint default boolean vmNameMustNotBeNull() { return vm().name() != null; } } }

* Enforce the constraints automatically. If the constraints are in the model, then we can just check them and reject requests before delivering them to the application. Currently we do this manually (and often forget) with calls to "validate(...)" methods.

Did you consider just annotating the DTOs with JSR-303 annotations and integrate a validator with jax-rs? See [2] for an example.

This is a great way to implement a system, but the goal here isn't to implement it, rather to specify it. Using annotations in this way won't help the generators of the SDKs, for example, to figure out what parameters are required, mandatory, etc.

* Generate the Java classes directly from the model. Instead of Model -> XML Schema -> Java, we can do Model -> Java. This will allow us to solve some of the XJC compiler limitations, like the horrible way we handle arrays today.

Swagger [3] is a rest documentation specification. There is also a maven plugin [4] and you can create clients for example with [5].

* Replace JAX-RS with a simpler infrastructure that supports better streaming and CDI injection.

With resteasy-cdi you have pretty good injection support for resteasy. Run the demo in [1] to see it in action and look at the file at [6].

Resteasy-CDI isn't standard, it only works with Resteasy. If we rely on it then we re tied to Resteasy for ever.

* Add support for multiple versions of the API, using the "Version" header, and generating different Java classes for entities and services. For example, if we have versions 4 and 5 of the model as separate artifacts, then we can generate "V4Vm" and "V5Vm" entity classes, and "V4VmService" and "V5VmService" service classes. These can be used simultaneously in the server, so we can have in the same engine implementations for multiple versions.

There are also many ways to do that. Here [7] is a pretty clean way to do it with jax-rs and you will have everything related in one resource.

Yes, there are many ways. In my opinion it is better to use the HTTP "Version" header, and to forward requests to different resource implementations without requiring different URLs or different content types.

The final picture isn't completely defined yet.

Regards, Juan Hernandez

> On Mon, Oct 26, 2015 at 4:03 PM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> wrote: > > Hello, > > I will soon merge the following patches that introduce a new way to > specify the contracts of the RESTAPI: > > restapi: Introduce metamodel > https://gerrit.ovirt.org/45852 > > restapi: Use metamodel > https://gerrit.ovirt.org/46478 > > restapi: Generate JAX-RS interfaces from model > https://gerrit.ovirt.org/47337 > >

> Looks pretty much like we are replacing one way of annotating things > with another way of specifying things. > Could you elaborate what the benefit of that way of description is? > > How would I customize endpoints with e.g. @Gzip annotations? Would I at > the end still have my JAX-RS annotates resource classes? > > > These patches introduce a new "metamodel" concept, and move the current > specification of the RESTAPI based on XML schema and JAX-RS interfaces > to a new "model" built on the new metamodel. > > > What does this mean for you in practical terms? Currently when you want > to introduce or modify one of the data types used by the RESTAPI you > start by modifying the XML schema. Once the patches are merged the XML > schema will never be touched, as it will be automatically generated from > the "model". For example, imagine that you need to add a new "color" > attribute to the "VM" entity. To do so with the new model you will have > to modify the following file, which is the specification of the "Vm" > entity, written as a Java interface: > > > https://gerrit.ovirt.org/#/c/46478/16/backend/manager/modules/restapi/model/... > > In that interface you will have to add a line like this: > > String color(); > > Note that this Java interface is just the specification of the entity, > it won't be used at all during runtime. Instead of that the XML schema > will be generated from it, and then Java will be generated from the XML > schema, as we do today (this will change in the future, but not yet). > > Same for the services. If you want to add a new "paint" action to the > "Vm" resource then you won't modify the JAX-RS interfaces, instead of > that you will modify the following file, which is the specification of > the "Vm" service, written as a Java interface: > > > https://gerrit.ovirt.org/#/c/47337/6/backend/manager/modules/restapi/model/s... > > In that interface you will need to add a sub-interface representing the > action: > > interface Paint { > } > > The JAX-RS interface will be generated from that. Currently these > sub-interfaces are empty. In the future they will contain the > specifications of the parameters (currently in the rsdl_metadata.yml > file). > > > > These changes will currently affect only the specification of the > RESTAPI, not the implementation, so in in the "Backend*Resource" classes > things won't change yet. > > > Currently I do not really understand where we are going here. Are we > trying to get rid of rdsl? > > So basically two questions: > > 1) What is the final goal? > 2) What speaks agains using Hibernate validator on Daos in combination > with JAX-RS annotated resources (and just removing all interfaces, as > far as I can see we only have one implementation per endpoint) and > creating all schemas and clients through SWAGGER tooling? > > > If you have doubts, please let me know. > > Regards, > Juan Hernandez > > -- > Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta > 3ºD, 28016 Madrid, Spain > Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat > S.L. > _______________________________________________ > Devel mailing list > Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > http://lists.ovirt.org/mailman/listinfo/devel > > > Thanks, > > Roman

-- Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta 3ºD, 28016 Madrid, Spain Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.

I don't know if it is the right thing to do to invent something new here. I personally would prefer to thread a path which is very common on the java community. I would love follow the DRY principle regarding to the stack and the code and would just use the great community projects there.

It would also completely eliminate any custom magic. The JAX-RS and CDI magic is pretty standard and easy to understand. From my perspective, real JAX-RS resoures have the advantage of

* being very easy to understand (there is magic, but the connection to the real endpoint is pretty clear) * being easy to customize suff, like adding @GZip to an annotation * describing pretty clearly the connection between the generated rest interface and the internal services

Finally writing hand crafted tests is also much easier.

What are your thoughts about that?

Best Regards, Roman

[1] https://github.com/rmohr/jetty-maven-cdi-demo [2] https://github.com/rmohr/jetty-maven-cdi-demo/blob/master/src/main/java/rmoh... [3] http://swagger.io/ [4] https://github.com/kongchen/swagger-maven-plugin [5] https://github.com/swagger-api/swagger-codegen [6] https://github.com/rmohr/jetty-maven-cdi-demo/blob/master/src/main/java/rmoh... [7] http://maxenglander.com/2013/04/23/basic-restful-api-versioning-in-jersey.ht... [8] https://github.com/swagger-api/swagger-ui

-- Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta 3ºD, 28016 Madrid, Spain Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.

On 10/27/2015 11:28 AM, Roman Mohr wrote:

On Tue, Oct 27, 2015 at 10:47 AM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com>> wrote:

On 10/27/2015 10:16 AM, Roman Mohr wrote: > > > On Mon, Oct 26, 2015 at 5:32 PM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> wrote: > > On 10/26/2015 04:56 PM, Roman Mohr wrote: > > Hi Juan, > > > > The way to specify the contract look pretty clean and nice. > > I would love to read a few words about the big picture. What is the > > final scenario? > > > > The motivation for this change is that currently we don't have a central > place where the RESTAPI is specified, rather we have several different > places, using several different technologies: > > * XML schema for the data model. > * JAX-RS for part of the operational model (without the parameters). > * rsdl_metadata.yaml for the parameters of the operational model. > > This makes it difficult to infer information about the model. For > example, the generators of the SDKs have to download the XML schema, and > the RSDL (which is generated from the JAX-RS interfaces using reflection > and combining it with the information from the rsdl_metadata.yaml file) > and then they have to do their own computations to extract what they > need. > > Same happens with the CLI: it has to extract the information it needs > from the Python code generated for the Python SDK, yet another level of > indirection. > > > You are right, that definitely needs to be cleaned up. I just want to > discuss a few points below with you. > > > > We are also lacking a comprehensive reference documentation of the > RESTAPI. What we currently have has been written by hand, and gets out > of sync very quickly, and we don't even notice. > > > Did you also consider swagger? It is made for exactly that purpose. > I created a demo in [1] which uses resteasy, weld, hibernate-validator > and swagger to demonstrate how to do DRY with jaxrs. > Would be great to hear you thoughts on that. > > And there is the great swagger-ui [8] to display the documentation in a > more human readable way. >

Yes, I considered Swagger, and rejected it because it is JSON centric, and I think JSON isn't as good as Java to represent the contracts of our RESTAPI.

You just write plain jax-rs, swagger just creates a description out of it. So the source defining the contract is pure java (jax-rs with some swagger annotations for description, etc.). Or am I missing the point here?

If I understand correctly the Swagger core is a JSON (or YAML) specification of the API. From that you can generate JAX-RS annotated code, not the other way around. So the specification document that you write is a JSON document. Alternatively, you can use the Swagger annotations to decorate your implementation, both the entity classes and the JAX-RS resource implementations, and then extract the model from that. But this is putting the implementation before the specification. That is where we are today, and it causes multiple problems. I think it is better to have the specification and the implementation separate. Swagger does that well when using JSON directly, our metamodel also does it well, but using a better language.

In addition we need to do these changes in a smooth way, without causing big changes in the middle. For example, in the first step we need to preserve the JAX-RS interfaces as they are today, to avoid massive changes to all the resource implementations. This could be done with

Swagger, but would require custom code generators. With less effort we can do our own.

This is of course generally a difficult task. But I do not know why it would be more difficult to write a custom swagger reader (if we even have to, it can read the interfaces as well) . They are pretty streight forward. Just look at [9], this contains the wole jax-rs specific code to generate the swagger documentation.

But yes, I don't know every detail here of the engine and can't clearly say that integrating that would just streight forward (my feeling tells me that it would not be too hard). I am just under the impression that we would benefit from that. Just reduces custom magic to a minimum.

Using something like Swagger would be certainly possible, and not that hard, but it requires an effort. For example, say that we decide to use the Swagger annotations. Then we will need to add these annotations to all our JAX-RS resource implementations. That is a non trivial effort. We would need to add the annotations to the entities as well. But wait, we don't have such entities, only XML schema. So we would need a reader that parses XML schema, and creating it requires effort. Where do we put the documentation then? Part in the JAX-RS interfaces, part in the XML schema. We are already there, and we ended up with no documentation at all. In my view the sum of these efforts is higher than doing our own metamodel.

Swagger UI is certainly great. I did test it and it is really good. We may be able to copy some concepts.

> > > To solve these issues I intend to have the specification of the RESTAPI > only in one place, and using only one technology. I decided to use Java > interfaces for that. Note however that they are just the support for the > information, like paper is the support for ink. I decided to use Java > because it is easy to create, modify and re-factor using tools familiar > to most of us. > > These source of these interfaces is analysed (using QDox, currently) and > a "model" of the RESTAPI is generated in memory. This model is > independent of the supporting Java source, and easy to consume. For > example, imagine that you want to list all the types available in the > model and for each one display its documentation: > > Model model = ...; > for (Type type : model.getTypes()) { > Name name = type.getName(); > String doc = type.getDoc(); > System.out.println(name + ": " + doc); > } > > Something like this, but more elaborate, will be part of a web > application that provides comprehensive reference documentation, > assuming that we dedicate the time to write documentation comments in > the specification. > > I intend to use this model also to do simplify the generators of the > SDKs and the CLI. > > In addition these are some of the things that I would like to change in > the near future (for 4.0): > > * Move the specification of the parameters of operations out of the > rsdl_metadata.yaml file and into the model. For example: > > @Service > public VmService { > /** > * The operation to add a virtual machine. > */ > interface Add { > /** > * The representation of the virtual machine is received > * as parameter, and the representation of the created > * virtual machine is returned as result. > */ > @In @Out Vm vm(); > > /** > * In the future, we will be able to specify other > * parameters here. > */ > @In Boolean force(); > > /** > * Even with default values. > */ > @In default Boolean force() { return true; } > > /** > * And we will be able to specify constraints, which > * will replace the rsdl_metadata.yaml file. > */ > @Constraint > default boolean vmNameMustNotBeNull() { > return vm().name() != null; > } > } > } > > * Enforce the constraints automatically. If the constraints are in the > model, then we can just check them and reject requests before delivering > them to the application. Currently we do this manually (and often > forget) with calls to "validate(...)" methods. > > > > Did you consider just annotating the DTOs with JSR-303 annotations and > integrate a validator with jax-rs? > See [2] for an example. >

This is a great way to implement a system, but the goal here isn't to implement it, rather to specify it. Using annotations in this way won't help the generators of the SDKs, for example, to figure out what parameters are required, mandatory, etc.

Swagger understands them. From my example project, swagger created that

description: type: "string" minLength: 10 maxLength: 100

out of

@Size(min=10, max=100) # jsr-303 private String description;

and so does swagger-codegen which can generate clients in java, python, ...

This is extracting the specification from the implementation, which isn't correct in my opinion, it should be the opposite. Not saying that this makes Swagger bad, it is nice that it has this capability, but I think we can do it better.

> > > * Generate the Java classes directly from the model. Instead of Model -> > XML Schema -> Java, we can do Model -> Java. This will allow us to solve > some of the XJC compiler limitations, like the horrible way we handle > arrays today. > > > Swagger [3] is a rest documentation specification. There is also a maven > plugin [4] and you can create clients for example with [5]. > > > > * Replace JAX-RS with a simpler infrastructure that supports better > streaming and CDI injection. > > > > With resteasy-cdi you have pretty good injection support for resteasy. > Run the demo in [1] to see it in action and look at the file at [6]. >

Resteasy-CDI isn't standard, it only works with Resteasy. If we rely on it then we re tied to Resteasy for ever.

Even jersey has support for that (I think it is called jeryse-gf-cdi), but why would we want switch? I don't think that jboss will drop resteasy and it also works fine outside of full blown containers. I don't think that this is an argument.

Well, nobody thought that JBoss would drop Tomcat, and they did. Nobody thought that Resteasy would change the SPI from 2.x to 3.x, and they did. We want (well, I want) to get out of JAX-RS because it doesn't support well CDI and streaming.

> > > * Add support for multiple versions of the API, using the "Version" > header, and generating different Java classes for entities and services. > For example, if we have versions 4 and 5 of the model as separate > artifacts, then we can generate "V4Vm" and "V5Vm" entity classes, and > "V4VmService" and "V5VmService" service classes. These can be used > simultaneously in the server, so we can have in the same engine > implementations for multiple versions. > > > There are also many ways to do that. Here [7] is a pretty clean way to > do it with jax-rs and you will have everything related in one resource. >

Yes, there are many ways. In my opinion it is better to use the HTTP "Version" header, and to forward requests to different resource implementations without requiring different URLs or different content types.

Have no strong opinion there, just seemed to be a good choice regarding to versioning limitations in jax-rs and our use of jax-rs subresources.

> > > The final picture isn't completely defined yet. > > Regards, > Juan Hernandez > > > On Mon, Oct 26, 2015 at 4:03 PM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>> wrote: > > > > Hello, > > > > I will soon merge the following patches that introduce a new > way to > > specify the contracts of the RESTAPI: > > > > restapi: Introduce metamodel > > https://gerrit.ovirt.org/45852 > > > > restapi: Use metamodel > > https://gerrit.ovirt.org/46478 > > > > restapi: Generate JAX-RS interfaces from model > > https://gerrit.ovirt.org/47337 > > > > > > > Looks pretty much like we are replacing one way of annotating things > > with another way of specifying things. > > Could you elaborate what the benefit of that way of description is? > > > > How would I customize endpoints with e.g. @Gzip annotations? Would > I at > > the end still have my JAX-RS annotates resource classes? > > > > > > These patches introduce a new "metamodel" concept, and move > the current > > specification of the RESTAPI based on XML schema and JAX-RS > interfaces > > to a new "model" built on the new metamodel. > > > > > > What does this mean for you in practical terms? Currently when > you want > > to introduce or modify one of the data types used by the > RESTAPI you > > start by modifying the XML schema. Once the patches are merged > the XML > > schema will never be touched, as it will be automatically > generated from > > the "model". For example, imagine that you need to add a new > "color" > > attribute to the "VM" entity. To do so with the new model you > will have > > to modify the following file, which is the specification of > the "Vm" > > entity, written as a Java interface: > > > > > > > https://gerrit.ovirt.org/#/c/46478/16/backend/manager/modules/restapi/model/... > > > > In that interface you will have to add a line like this: > > > > String color(); > > > > Note that this Java interface is just the specification of the > entity, > > it won't be used at all during runtime. Instead of that the > XML schema > > will be generated from it, and then Java will be generated > from the XML > > schema, as we do today (this will change in the future, but > not yet). > > > > Same for the services. If you want to add a new "paint" action > to the > > "Vm" resource then you won't modify the JAX-RS interfaces, > instead of > > that you will modify the following file, which is the > specification of > > the "Vm" service, written as a Java interface: > > > > > > > https://gerrit.ovirt.org/#/c/47337/6/backend/manager/modules/restapi/model/s... > > > > In that interface you will need to add a sub-interface > representing the > > action: > > > > interface Paint { > > } > > > > The JAX-RS interface will be generated from that. Currently these > > sub-interfaces are empty. In the future they will contain the > > specifications of the parameters (currently in the > rsdl_metadata.yml > > file). > > > > > > > > These changes will currently affect only the specification of the > > RESTAPI, not the implementation, so in in the > "Backend*Resource" classes > > things won't change yet. > > > > > > Currently I do not really understand where we are going here. Are we > > trying to get rid of rdsl? > > > > So basically two questions: > > > > 1) What is the final goal? > > 2) What speaks agains using Hibernate validator on Daos in combination > > with JAX-RS annotated resources (and just removing all interfaces, as > > far as I can see we only have one implementation per endpoint) and > > creating all schemas and clients through SWAGGER tooling? > > > > > > If you have doubts, please let me know. > > > > Regards, > > Juan Hernandez > > > > -- > > Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, > planta > > 3ºD, 28016 Madrid, Spain > > Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - > Red Hat > > S.L. > > _______________________________________________ > > Devel mailing list > > Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>> > > http://lists.ovirt.org/mailman/listinfo/devel > > > > > > Thanks, > > > > Roman > > > -- > Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta > 3ºD, 28016 Madrid, Spain > Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat > S.L. > > > > I don't know if it is the right thing to do to invent something new > here. I personally would prefer to thread a path which is very common on > the java community. > I would love follow the DRY principle regarding to the stack and the > code and would just use the great community projects there. > > It would also completely eliminate any custom magic. The JAX-RS and CDI > magic is pretty standard and easy to understand. > From my perspective, real JAX-RS resoures have the advantage of > > * being very easy to understand (there is magic, but the connection to > the real endpoint is pretty clear) > * being easy to customize suff, like adding @GZip to an annotation > * describing pretty clearly the connection between the generated rest > interface and the internal services > > Finally writing hand crafted tests is also much easier. > > What are your thoughts about that? > > Best Regards, > Roman > > > [1] https://github.com/rmohr/jetty-maven-cdi-demo > [2] > https://github.com/rmohr/jetty-maven-cdi-demo/blob/master/src/main/java/rmoh... > [3] http://swagger.io/ > [4] https://github.com/kongchen/swagger-maven-plugin > [5] https://github.com/swagger-api/swagger-codegen > [6] > https://github.com/rmohr/jetty-maven-cdi-demo/blob/master/src/main/java/rmoh... > [7] > http://maxenglander.com/2013/04/23/basic-restful-api-versioning-in-jersey.ht... > [8] https://github.com/swagger-api/swagger-ui >

[9] https://github.com/swagger-api/swagger-core/blob/master/modules/swagger-jaxr...

-- Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta 3ºD, 28016 Madrid, Spain Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.

On 10/27/2015 12:55 PM, Roman Mohr wrote:

On Tue, Oct 27, 2015 at 12:16 PM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com>> wrote:

On 10/27/2015 11:28 AM, Roman Mohr wrote: > > > On Tue, Oct 27, 2015 at 10:47 AM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> wrote: > > On 10/27/2015 10:16 AM, Roman Mohr wrote: > > > > > > On Mon, Oct 26, 2015 at 5:32 PM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>> wrote: > > > > On 10/26/2015 04:56 PM, Roman Mohr wrote: > > > Hi Juan, > > > > > > The way to specify the contract look pretty clean and nice. > > > I would love to read a few words about the big picture. What > is the > > > final scenario? > > > > > > > The motivation for this change is that currently we don't have > a central > > place where the RESTAPI is specified, rather we have several > different > > places, using several different technologies: > > > > * XML schema for the data model. > > * JAX-RS for part of the operational model (without the > parameters). > > * rsdl_metadata.yaml for the parameters of the operational model. > > > > This makes it difficult to infer information about the model. For > > example, the generators of the SDKs have to download the XML > schema, and > > the RSDL (which is generated from the JAX-RS interfaces using > reflection > > and combining it with the information from the > rsdl_metadata.yaml file) > > and then they have to do their own computations to extract > what they > > need. > > > > Same happens with the CLI: it has to extract the information > it needs > > from the Python code generated for the Python SDK, yet another > level of > > indirection. > > > > > > You are right, that definitely needs to be cleaned up. I just want to > > discuss a few points below with you. > > > > > > > > We are also lacking a comprehensive reference documentation of the > > RESTAPI. What we currently have has been written by hand, and > gets out > > of sync very quickly, and we don't even notice. > > > > > > Did you also consider swagger? It is made for exactly that purpose. > > I created a demo in [1] which uses resteasy, weld, hibernate-validator > > and swagger to demonstrate how to do DRY with jaxrs. > > Would be great to hear you thoughts on that. > > > > And there is the great swagger-ui [8] to display the documentation > in a > > more human readable way. > > > > Yes, I considered Swagger, and rejected it because it is JSON centric, > and I think JSON isn't as good as Java to represent the contracts of our > RESTAPI. > > > You just write plain jax-rs, swagger just creates a description out of > it. So the source defining the contract is pure java (jax-rs with some > swagger annotations for description, etc.). > Or am I missing the point here? >

If I understand correctly the Swagger core is a JSON (or YAML) specification of the API. From that you can generate JAX-RS annotated code, not the other way around. So the specification document that you write is a JSON document.

You are right, my terminology here was not clear. Swagger is just a specification. Swagger-core and swagger-jaxrs are the ones which can create the documnetation out of JAX-RS resources.

Alternatively, you can use the Swagger annotations to decorate your implementation, both the entity classes and the JAX-RS resource implementations, and then extract the model from that. But this is putting the implementation before the specification. That is where we are today, and it causes multiple problems. I think it is better to have the specification and the implementation separate. Swagger does that well when using JSON directly, our metamodel also does it well, but using a better language.

Isn't our problem that we have everything scattered arount the place and not that we are using JAX-RS? I don't think that this has anything to do with specification before implementation or implementation before specification.

Correct, the problem with the specification if that we don't have one, just pieces of it scattered around, some in the XML schema, some in the JAX-RS interfaces, some in the rsdl_metadata.yaml file. This is historically related with the order of specification and implementation. When the project started we didn't create a specification of the RESTAPI, only an implementation, using JAX-RS and XML schema. Then we added the rsdl_metadata.yaml file, and extracted a specification from that, the RSDL. The result is hard to consume. To make it easy to consume we need to reverse the order: first specification, free of implementation details, then implementation.

> > > In addition we need to do these changes in a smooth way, without causing > big changes in the middle. For example, in the first step we need to > preserve the JAX-RS interfaces as they are today, to avoid massive > changes to all the resource implementations. This could be done with > > Swagger, but would require custom code generators. With less effort we > can do our own. > > > This is of course generally a difficult task. But I do not know why it > would be more difficult to write a custom swagger reader (if we even > have to, it can read the interfaces as well) . > They are pretty streight forward. Just look at [9], this contains the > wole jax-rs specific code to generate the swagger documentation. > > But yes, I don't know every detail here of the engine and can't clearly > say that integrating that would just streight forward (my feeling tells > me that it would not be too hard). I am just under the impression that > we would benefit from that. Just reduces custom magic to a minimum. >

Using something like Swagger would be certainly possible, and not that hard, but it requires an effort. For example, say that we decide to use the Swagger annotations. Then we will need to add these annotations to all our JAX-RS resource implementations. That is a non trivial effort. We would need to add the annotations to the entities as well. But wait, we don't have such entities, only XML schema. So we would need a reader that parses XML schema, and creating it requires effort.

You can just create the entities/daos once like we do now on every build and annotate them once and drop the whole xml.

The DAOs aren't part of the RESTAPI. The backend entities aren't either. The RESTAPI has its own entities which are currently generated from XML schema.

Where do we put the documentation then? Part in the JAX-RS interfaces, part in the XML schema.

I don't understand that. There is only one documentation and implementation source, that is the JAX-RS resource and the entity/dao accepted by the endpoint. The XML schema can just be removed. The documentation in swagger format for other tools like swagger-codegen and swagger-ci can be made available through swagger-maven-plugin or a servlet in the engine which generates the swagger json on the fly (like I did in [1])

Those are two: JAX-RS and entities, and they are implementations, not specifications. The XML schema can't be drop, as many clients use it. But we can generate it from the model, and that is what we will do.

We are already there, and we ended up with no documentation at all. In my view the sum of these efforts is higher than doing our own metamodel.

> > > Swagger UI is certainly great. I did test it and it is really good. We > may be able to copy some concepts. > > > > > > > To solve these issues I intend to have the specification of > the RESTAPI > > only in one place, and using only one technology. I decided to > use Java > > interfaces for that. Note however that they are just the > support for the > > information, like paper is the support for ink. I decided to > use Java > > because it is easy to create, modify and re-factor using tools > familiar > > to most of us. > > > > These source of these interfaces is analysed (using QDox, > currently) and > > a "model" of the RESTAPI is generated in memory. This model is > > independent of the supporting Java source, and easy to > consume. For > > example, imagine that you want to list all the types available > in the > > model and for each one display its documentation: > > > > Model model = ...; > > for (Type type : model.getTypes()) { > > Name name = type.getName(); > > String doc = type.getDoc(); > > System.out.println(name + ": " + doc); > > } > > > > Something like this, but more elaborate, will be part of a web > > application that provides comprehensive reference documentation, > > assuming that we dedicate the time to write documentation > comments in > > the specification. > > > > I intend to use this model also to do simplify the generators > of the > > SDKs and the CLI. > > > > In addition these are some of the things that I would like to > change in > > the near future (for 4.0): > > > > * Move the specification of the parameters of operations out > of the > > rsdl_metadata.yaml file and into the model. For example: > > > > @Service > > public VmService { > > /** > > * The operation to add a virtual machine. > > */ > > interface Add { > > /** > > * The representation of the virtual machine is received > > * as parameter, and the representation of the created > > * virtual machine is returned as result. > > */ > > @In @Out Vm vm(); > > > > /** > > * In the future, we will be able to specify other > > * parameters here. > > */ > > @In Boolean force(); > > > > /** > > * Even with default values. > > */ > > @In default Boolean force() { return true; } > > > > /** > > * And we will be able to specify constraints, which > > * will replace the rsdl_metadata.yaml file. > > */ > > @Constraint > > default boolean vmNameMustNotBeNull() { > > return vm().name() != null; > > } > > } > > } > > > > * Enforce the constraints automatically. If the constraints > are in the > > model, then we can just check them and reject requests before > delivering > > them to the application. Currently we do this manually (and often > > forget) with calls to "validate(...)" methods. > > > > > > > > Did you consider just annotating the DTOs with JSR-303 annotations and > > integrate a validator with jax-rs? > > See [2] for an example. > > > > This is a great way to implement a system, but the goal here isn't to > implement it, rather to specify it. Using annotations in this way won't > help the generators of the SDKs, for example, to figure out what > parameters are required, mandatory, etc. > > > Swagger understands them. From my example project, swagger created that > > description: > type: "string" > minLength: 10 > maxLength: 100 > > out of > > @Size(min=10, max=100) # jsr-303 > private String description; > > and so does swagger-codegen which can generate clients in java, python, ... >

This is extracting the specification from the implementation, which isn't correct in my opinion, it should be the opposite. Not saying that this makes Swagger bad, it is nice that it has this capability, but I think we can do it better.

In my opition this is the main advantage of that. It is DRY while still having full control of the implementation.

I think that DRY is better served if you write the specification once, and then, from that, you generate the contracts (interfaces, entities, builders, etc) that the implementation should use.

> > > > > > * Generate the Java classes directly from the model. Instead of Model -> > > XML Schema -> Java, we can do Model -> Java. This will allow us to solve > > some of the XJC compiler limitations, like the horrible way we handle > > arrays today. > > > > > > Swagger [3] is a rest documentation specification. There is also a maven > > plugin [4] and you can create clients for example with [5]. > > > > > > > > * Replace JAX-RS with a simpler infrastructure that supports better > > streaming and CDI injection. > > > > > > > > With resteasy-cdi you have pretty good injection support for resteasy. > > Run the demo in [1] to see it in action and look at the file at [6]. > > > > Resteasy-CDI isn't standard, it only works with Resteasy. If we rely on > it then we re tied to Resteasy for ever. > > > Even jersey has support for that (I think it is called jeryse-gf-cdi), > but why would we want switch? I don't think that jboss will drop > resteasy and it also works fine outside of full blown containers. I > don't think that this is an argument. >

Well, nobody thought that JBoss would drop Tomcat, and they did. Nobody thought that Resteasy would change the SPI from 2.x to 3.x, and they did.\

It will be there for the next year or another library which offers the same thing. Having injections in Jax-rs resources is important for spring, jboss, glassfish and others there will always be ways to do that. I don't know why we should create our own 'custom standard' just because another standard does not include dependency injection when we can just extend it so easily.

Here our views of "easily" are different. I have suffered the problems with changes in Resteasy over the past years, and I won't describe it as easy.

We want (well, I want) to get out of JAX-RS because it doesn't support well CDI and streaming.

Why should streaming resources be in the rest interface?

Streaming should be used not only in the RESTAPI, but in the complete system, including the backend and the DAL, otherwise we have serious problems when clients requests large amounts of objects. Think, for example, that a client requests 100 virtual machines. What we currently do is the following: * The DAL requests the 100 rows to the database, and waits till the 100 rows are loaded in memory (this is how the JDBC driver works by default). Only when the 100 rows are in memory the are converted to backend objects. * The DAL returns the 100 backend objects to the BLL, which processes them. Only when they are all processed they are returned to the RESTAPI, as a list. * The RESTAPI transforms the 100 backend objects into RESTAPI objects. Only when the are all transformed they are returned by the corresponding JAX-RS method to the JAX-RS infrastructure. This is encouraged by the high level interface of JAX-RS, as it encourages you to write methods like this: @GET Vms get() * Once the JAX-RS infrastructure has the complete result it starts to convert it to the XML (or JSON) document. Same for the reverse direction. Put 10K virtual machines instead of just 100 and you see the problem. Of course one could use JAX-RS in a way that doesn't have this problem, it is certainly possible, you can get the output stream and serialize the result from the resource implementation, in a an streaming fashion. But then the value of JAX-RS is reduced, you can do the same with a simpler servlet architecture. That is what I plan to do.

> > > > > > > * Add support for multiple versions of the API, using the "Version" > > header, and generating different Java classes for entities and services. > > For example, if we have versions 4 and 5 of the model as separate > > artifacts, then we can generate "V4Vm" and "V5Vm" entity classes, and > > "V4VmService" and "V5VmService" service classes. These can be used > > simultaneously in the server, so we can have in the same engine > > implementations for multiple versions. > > > > > > There are also many ways to do that. Here [7] is a pretty clean way to > > do it with jax-rs and you will have everything related in one resource. > > > > Yes, there are many ways. In my opinion it is better to use the HTTP > "Version" header, and to forward requests to different resource > implementations without requiring different URLs or different > content types. > > Have no strong opinion there, just seemed to be a good choice regarding > to versioning limitations in jax-rs and our use of jax-rs subresources. > > > > > > > > The final picture isn't completely defined yet. > > > > Regards, > > Juan Hernandez > > > > > On Mon, Oct 26, 2015 at 4:03 PM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> > > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>>> wrote: > > > > > > Hello, > > > > > > I will soon merge the following patches that introduce a new > > way to > > > specify the contracts of the RESTAPI: > > > > > > restapi: Introduce metamodel > > > https://gerrit.ovirt.org/45852 > > > > > > restapi: Use metamodel > > > https://gerrit.ovirt.org/46478 > > > > > > restapi: Generate JAX-RS interfaces from model > > > https://gerrit.ovirt.org/47337 > > > > > > > > > > > Looks pretty much like we are replacing one way of > annotating things > > > with another way of specifying things. > > > Could you elaborate what the benefit of that way of > description is? > > > > > > How would I customize endpoints with e.g. @Gzip annotations? > Would > > I at > > > the end still have my JAX-RS annotates resource classes? > > > > > > > > > These patches introduce a new "metamodel" concept, and move > > the current > > > specification of the RESTAPI based on XML schema and JAX-RS > > interfaces > > > to a new "model" built on the new metamodel. > > > > > > > > > What does this mean for you in practical terms? > Currently when > > you want > > > to introduce or modify one of the data types used by the > > RESTAPI you > > > start by modifying the XML schema. Once the patches are > merged > > the XML > > > schema will never be touched, as it will be automatically > > generated from > > > the "model". For example, imagine that you need to add a new > > "color" > > > attribute to the "VM" entity. To do so with the new > model you > > will have > > > to modify the following file, which is the specification of > > the "Vm" > > > entity, written as a Java interface: > > > > > > > > > > > > https://gerrit.ovirt.org/#/c/46478/16/backend/manager/modules/restapi/model/... > > > > > > In that interface you will have to add a line like this: > > > > > > String color(); > > > > > > Note that this Java interface is just the specification > of the > > entity, > > > it won't be used at all during runtime. Instead of that the > > XML schema > > > will be generated from it, and then Java will be generated > > from the XML > > > schema, as we do today (this will change in the future, but > > not yet). > > > > > > Same for the services. If you want to add a new "paint" > action > > to the > > > "Vm" resource then you won't modify the JAX-RS interfaces, > > instead of > > > that you will modify the following file, which is the > > specification of > > > the "Vm" service, written as a Java interface: > > > > > > > > > > > > https://gerrit.ovirt.org/#/c/47337/6/backend/manager/modules/restapi/model/s... > > > > > > In that interface you will need to add a sub-interface > > representing the > > > action: > > > > > > interface Paint { > > > } > > > > > > The JAX-RS interface will be generated from that. > Currently these > > > sub-interfaces are empty. In the future they will > contain the > > > specifications of the parameters (currently in the > > rsdl_metadata.yml > > > file). > > > > > > > > > > > > These changes will currently affect only the > specification of the > > > RESTAPI, not the implementation, so in in the > > "Backend*Resource" classes > > > things won't change yet. > > > > > > > > > Currently I do not really understand where we are going > here. Are we > > > trying to get rid of rdsl? > > > > > > So basically two questions: > > > > > > 1) What is the final goal? > > > 2) What speaks agains using Hibernate validator on Daos in > combination > > > with JAX-RS annotated resources (and just removing all > interfaces, as > > > far as I can see we only have one implementation per > endpoint) and > > > creating all schemas and clients through SWAGGER tooling? > > > > > > > > > If you have doubts, please let me know. > > > > > > Regards, > > > Juan Hernandez > > > > > > -- > > > Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. > Gorbea 3, > > planta > > > 3ºD, 28016 Madrid, Spain > > > Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - > > Red Hat > > > S.L. > > > _______________________________________________ > > > Devel mailing list > > > Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>> > > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>>> > > > http://lists.ovirt.org/mailman/listinfo/devel > > > > > > > > > Thanks, > > > > > > Roman > > > > > > -- > > Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, > planta > > 3ºD, 28016 Madrid, Spain > > Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - > Red Hat > > S.L. > > > > > > > > I don't know if it is the right thing to do to invent something new > > here. I personally would prefer to thread a path which is very > common on > > the java community. > > I would love follow the DRY principle regarding to the stack and the > > code and would just use the great community projects there. > > > > It would also completely eliminate any custom magic. The JAX-RS > and CDI > > magic is pretty standard and easy to understand. > > From my perspective, real JAX-RS resoures have the advantage of > > > > * being very easy to understand (there is magic, but the > connection to > > the real endpoint is pretty clear) > > * being easy to customize suff, like adding @GZip to an annotation > > * describing pretty clearly the connection between the generated rest > > interface and the internal services > > > > Finally writing hand crafted tests is also much easier. > > > > What are your thoughts about that? > > > > Best Regards, > > Roman > > > > > > [1] https://github.com/rmohr/jetty-maven-cdi-demo > > [2] > > > https://github.com/rmohr/jetty-maven-cdi-demo/blob/master/src/main/java/rmoh... > > [3] http://swagger.io/ > > [4] https://github.com/kongchen/swagger-maven-plugin > > [5] https://github.com/swagger-api/swagger-codegen > > [6] > > > https://github.com/rmohr/jetty-maven-cdi-demo/blob/master/src/main/java/rmoh... > > [7] > > > http://maxenglander.com/2013/04/23/basic-restful-api-versioning-in-jersey.ht... > > [8] https://github.com/swagger-api/swagger-ui > > > > [9] > https://github.com/swagger-api/swagger-core/blob/master/modules/swagger-jaxr... >

-- Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta 3ºD, 28016 Madrid, Spain Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.

On 10/27/2015 01:59 PM, Martin Betak wrote:

----- Original Message -----

...

From: "Juan Hernández" <jhernand@redhat.com> To: "Roman Mohr" <rmohr@redhat.com> Cc: devel@ovirt.org Sent: Tuesday, October 27, 2015 1:45:58 PM Subject: Re: [ovirt-devel] [ATN] Introduction of RESTAPI metamodel

On 10/27/2015 12:55 PM, Roman Mohr wrote:

...

On Tue, Oct 27, 2015 at 12:16 PM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com>> wrote:

On 10/27/2015 11:28 AM, Roman Mohr wrote: > > > On Tue, Oct 27, 2015 at 10:47 AM, Juan Hernández <jhernand@redhat.com > <mailto:jhernand@redhat.com> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> wrote: > > On 10/27/2015 10:16 AM, Roman Mohr wrote: > > > > > > On Mon, Oct 26, 2015 at 5:32 PM, Juan Hernández > > <jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>> wrote: > > > > On 10/26/2015 04:56 PM, Roman Mohr wrote: > > > Hi Juan, > > > > > > The way to specify the contract look pretty clean and nice. > > > I would love to read a few words about the big picture. What > is the > > > final scenario? > > > > > > > The motivation for this change is that currently we don't have > a central > > place where the RESTAPI is specified, rather we have > > several > different > > places, using several different technologies: > > > > * XML schema for the data model. > > * JAX-RS for part of the operational model (without the > parameters). > > * rsdl_metadata.yaml for the parameters of the operational model. > > > > This makes it difficult to infer information about the model. For > > example, the generators of the SDKs have to download the > > XML > schema, and > > the RSDL (which is generated from the JAX-RS interfaces using > reflection > > and combining it with the information from the > rsdl_metadata.yaml file) > > and then they have to do their own computations to extract > what they > > need. > > > > Same happens with the CLI: it has to extract the > > information > it needs > > from the Python code generated for the Python SDK, yet another > level of > > indirection. > > > > > > You are right, that definitely needs to be cleaned up. I just want to > > discuss a few points below with you. > > > > > > > > We are also lacking a comprehensive reference documentation of the > > RESTAPI. What we currently have has been written by hand, and > gets out > > of sync very quickly, and we don't even notice. > > > > > > Did you also consider swagger? It is made for exactly that purpose. > > I created a demo in [1] which uses resteasy, weld, hibernate-validator > > and swagger to demonstrate how to do DRY with jaxrs. > > Would be great to hear you thoughts on that. > > > > And there is the great swagger-ui [8] to display the documentation > in a > > more human readable way. > > > > Yes, I considered Swagger, and rejected it because it is JSON centric, > and I think JSON isn't as good as Java to represent the contracts of our > RESTAPI. > > > You just write plain jax-rs, swagger just creates a description out > of > it. So the source defining the contract is pure java (jax-rs with some > swagger annotations for description, etc.). > Or am I missing the point here? >

If I understand correctly the Swagger core is a JSON (or YAML) specification of the API. From that you can generate JAX-RS annotated code, not the other way around. So the specification document that you write is a JSON document.

You are right, my terminology here was not clear. Swagger is just a specification. Swagger-core and swagger-jaxrs are the ones which can create the documnetation out of JAX-RS resources.

Alternatively, you can use the Swagger annotations to decorate your implementation, both the entity classes and the JAX-RS resource implementations, and then extract the model from that. But this is putting the implementation before the specification. That is where we are today, and it causes multiple problems. I think it is better to have the specification and the implementation separate. Swagger does that well when using JSON directly, our metamodel also does it well, but using a better language.

Isn't our problem that we have everything scattered arount the place and not that we are using JAX-RS? I don't think that this has anything to do with specification before implementation or implementation before specification.

Correct, the problem with the specification if that we don't have one, just pieces of it scattered around, some in the XML schema, some in the JAX-RS interfaces, some in the rsdl_metadata.yaml file.

This is historically related with the order of specification and implementation. When the project started we didn't create a specification of the RESTAPI, only an implementation, using JAX-RS and XML schema. Then we added the rsdl_metadata.yaml file, and extracted a specification from that, the RSDL. The result is hard to consume. To make it easy to consume we need to reverse the order: first specification, free of implementation details, then implementation.

...

> > > In addition we need to do these changes in a smooth way, without > causing > big changes in the middle. For example, in the first step we need > to > preserve the JAX-RS interfaces as they are today, to avoid > massive > changes to all the resource implementations. This could be done > with > > Swagger, but would require custom code generators. With less > effort we > can do our own. > > > This is of course generally a difficult task. But I do not know why > it > would be more difficult to write a custom swagger reader (if we even > have to, it can read the interfaces as well) . > They are pretty streight forward. Just look at [9], this contains the > wole jax-rs specific code to generate the swagger documentation. > > But yes, I don't know every detail here of the engine and can't > clearly > say that integrating that would just streight forward (my feeling > tells > me that it would not be too hard). I am just under the impression > that > we would benefit from that. Just reduces custom magic to a minimum. >

Using something like Swagger would be certainly possible, and not that hard, but it requires an effort. For example, say that we decide to use the Swagger annotations. Then we will need to add these annotations to all our JAX-RS resource implementations. That is a non trivial effort. We would need to add the annotations to the entities as well. But wait, we don't have such entities, only XML schema. So we would need a reader that parses XML schema, and creating it requires effort.

You can just create the entities/daos once like we do now on every build and annotate them once and drop the whole xml.

The DAOs aren't part of the RESTAPI. The backend entities aren't either. The RESTAPI has its own entities which are currently generated from XML schema.

...

Where do we put the documentation then? Part in the JAX-RS interfaces, part in the XML schema.

I don't understand that. There is only one documentation and implementation source, that is the JAX-RS resource and the entity/dao accepted by the endpoint. The XML schema can just be removed. The documentation in swagger format for other tools like swagger-codegen and swagger-ci can be made available through swagger-maven-plugin or a servlet in the engine which generates the swagger json on the fly (like I did in [1])

Those are two: JAX-RS and entities, and they are implementations, not specifications.

The XML schema can't be drop, as many clients use it. But we can generate it from the model, and that is what we will do.

...

We are already there, and we ended up with no documentation at all. In my view the sum of these efforts is higher than doing our own metamodel.

> > > Swagger UI is certainly great. I did test it and it is really good. We > may be able to copy some concepts. > > > > > > > To solve these issues I intend to have the specification of > the RESTAPI > > only in one place, and using only one technology. I decided to > use Java > > interfaces for that. Note however that they are just the > support for the > > information, like paper is the support for ink. I decided > > to > use Java > > because it is easy to create, modify and re-factor using tools > familiar > > to most of us. > > > > These source of these interfaces is analysed (using QDox, > currently) and > > a "model" of the RESTAPI is generated in memory. This model is > > independent of the supporting Java source, and easy to > consume. For > > example, imagine that you want to list all the types available > in the > > model and for each one display its documentation: > > > > Model model = ...; > > for (Type type : model.getTypes()) { > > Name name = type.getName(); > > String doc = type.getDoc(); > > System.out.println(name + ": " + doc); > > } > > > > Something like this, but more elaborate, will be part of a web > > application that provides comprehensive reference documentation, > > assuming that we dedicate the time to write documentation > comments in > > the specification. > > > > I intend to use this model also to do simplify the generators > of the > > SDKs and the CLI. > > > > In addition these are some of the things that I would like to > change in > > the near future (for 4.0): > > > > * Move the specification of the parameters of operations > > out > of the > > rsdl_metadata.yaml file and into the model. For example: > > > > @Service > > public VmService { > > /** > > * The operation to add a virtual machine. > > */ > > interface Add { > > /** > > * The representation of the virtual machine is received > > * as parameter, and the representation of the > > created > > * virtual machine is returned as result. > > */ > > @In @Out Vm vm(); > > > > /** > > * In the future, we will be able to specify other > > * parameters here. > > */ > > @In Boolean force(); > > > > /** > > * Even with default values. > > */ > > @In default Boolean force() { return true; } > > > > /** > > * And we will be able to specify constraints, which > > * will replace the rsdl_metadata.yaml file. > > */ > > @Constraint > > default boolean vmNameMustNotBeNull() { > > return vm().name() != null; > > } > > } > > } > > > > * Enforce the constraints automatically. If the constraints > are in the > > model, then we can just check them and reject requests before > delivering > > them to the application. Currently we do this manually (and often > > forget) with calls to "validate(...)" methods. > > > > > > > > Did you consider just annotating the DTOs with JSR-303 annotations and > > integrate a validator with jax-rs? > > See [2] for an example. > > > > This is a great way to implement a system, but the goal here isn't to > implement it, rather to specify it. Using annotations in this way won't > help the generators of the SDKs, for example, to figure out what > parameters are required, mandatory, etc. > > > Swagger understands them. From my example project, swagger created that > > description: > type: "string" > minLength: 10 > maxLength: 100 > > out of > > @Size(min=10, max=100) # jsr-303 > private String description; > > and so does swagger-codegen which can generate clients in java, python, ... >

This is extracting the specification from the implementation, which isn't correct in my opinion, it should be the opposite. Not saying that this makes Swagger bad, it is nice that it has this capability, but I think we can do it better.

In my opition this is the main advantage of that. It is DRY while still having full control of the implementation.

I think that DRY is better served if you write the specification once, and then, from that, you generate the contracts (interfaces, entities, builders, etc) that the implementation should use.

...

> > > > > > * Generate the Java classes directly from the model. > > Instead of Model -> > > XML Schema -> Java, we can do Model -> Java. This will > > allow us to solve > > some of the XJC compiler limitations, like the horrible way > > we handle > > arrays today. > > > > > > Swagger [3] is a rest documentation specification. There is > > also a maven > > plugin [4] and you can create clients for example with [5]. > > > > > > > > * Replace JAX-RS with a simpler infrastructure that > > supports better > > streaming and CDI injection. > > > > > > > > With resteasy-cdi you have pretty good injection support for > > resteasy. > > Run the demo in [1] to see it in action and look at the file at > > [6]. > > > > Resteasy-CDI isn't standard, it only works with Resteasy. If we > rely on > it then we re tied to Resteasy for ever. > > > Even jersey has support for that (I think it is called > jeryse-gf-cdi), > but why would we want switch? I don't think that jboss will drop > resteasy and it also works fine outside of full blown containers. I > don't think that this is an argument. >

Well, nobody thought that JBoss would drop Tomcat, and they did. Nobody thought that Resteasy would change the SPI from 2.x to 3.x, and they did.\

It will be there for the next year or another library which offers the same thing. Having injections in Jax-rs resources is important for spring, jboss, glassfish and others there will always be ways to do that. I don't know why we should create our own 'custom standard' just because another standard does not include dependency injection when we can just extend it so easily.

Here our views of "easily" are different. I have suffered the problems with changes in Resteasy over the past years, and I won't describe it as easy.

...

We want (well, I want) to get out of JAX-RS because it doesn't support well CDI and streaming.

Why should streaming resources be in the rest interface?

Streaming should be used not only in the RESTAPI, but in the complete system, including the backend and the DAL, otherwise we have serious problems when clients requests large amounts of objects.

Think, for example, that a client requests 100 virtual machines. What we currently do is the following:

* The DAL requests the 100 rows to the database, and waits till the 100 rows are loaded in memory (this is how the JDBC driver works by default). Only when the 100 rows are in memory the are converted to backend objects.

* The DAL returns the 100 backend objects to the BLL, which processes them. Only when they are all processed they are returned to the RESTAPI, as a list.

* The RESTAPI transforms the 100 backend objects into RESTAPI objects. Only when the are all transformed they are returned by the corresponding JAX-RS method to the JAX-RS infrastructure. This is encouraged by the high level interface of JAX-RS, as it encourages you to write methods like this:

@GET Vms get()

* Once the JAX-RS infrastructure has the complete result it starts to convert it to the XML (or JSON) document.

Same for the reverse direction.

Put 10K virtual machines instead of just 100 and you see the problem.

Of course one could use JAX-RS in a way that doesn't have this problem, it is certainly possible, you can get the output stream and serialize the result from the resource implementation, in a an streaming fashion. But then the value of JAX-RS is reduced, you can do the same with a simpler servlet architecture. That is what I plan to do.

Hi Juan,

First of all I have to say I find this meta-model a very nice idea and clever usage of java 8 interfaces - I hope we'll see more usage of java 8 in the engine like this.

I also like the idea of streaming the results. I'm sure this will help the overall performance.

But on the topic of performance: with this new meta-model will we gain the ability to specify (semi-) automatic aggregations in the meta-model? In my opinion PUSH and useful aggregations are what is mostly lacking in our current REST API in terms of usability by frontends or other systems like you can see in this bug from ManageIQ [1].

Thank you and best regards,

Martin

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1263357

The metamodel won't directly help in this area, although having an easy to consume description of the RESTAPI could be useful to advance in that direction. For example, currently there isn't a simple way to find out what aggregations are possible. Lets take the VM entity as an example, how do you know what can be aggregated? With the metamodel this is easy to find, because the concept of "links" between entities (and between services) is explicit: Model model = ...; Type vm = model.getType(NameParser.parseUsingCase("Vm")); for (Link link : vm.getLinks()) { ... } This will tell you that from a VM you can navigate to "disks", "nics", etc, so they are potential candidates for aggregation.

...

...

> > > > > > > * Add support for multiple versions of the API, using the "Version" > > header, and generating different Java classes for entities and services. > > For example, if we have versions 4 and 5 of the model as separate > > artifacts, then we can generate "V4Vm" and "V5Vm" entity classes, and > > "V4VmService" and "V5VmService" service classes. These can be used > > simultaneously in the server, so we can have in the same engine > > implementations for multiple versions. > > > > > > There are also many ways to do that. Here [7] is a pretty clean way to > > do it with jax-rs and you will have everything related in one resource. > > > > Yes, there are many ways. In my opinion it is better to use the HTTP > "Version" header, and to forward requests to different resource > implementations without requiring different URLs or different > content types. > > Have no strong opinion there, just seemed to be a good choice regarding > to versioning limitations in jax-rs and our use of jax-rs subresources. > > > > > > > > The final picture isn't completely defined yet. > > > > Regards, > > Juan Hernandez > > > > > On Mon, Oct 26, 2015 at 4:03 PM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> > > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>>> wrote: > > > > > > Hello, > > > > > > I will soon merge the following patches that introduce a new > > way to > > > specify the contracts of the RESTAPI: > > > > > > restapi: Introduce metamodel > > > https://gerrit.ovirt.org/45852 > > > > > > restapi: Use metamodel > > > https://gerrit.ovirt.org/46478 > > > > > > restapi: Generate JAX-RS interfaces from model > > > https://gerrit.ovirt.org/47337 > > > > > > > > > > > Looks pretty much like we are replacing one way of > annotating things > > > with another way of specifying things. > > > Could you elaborate what the benefit of that way of > description is? > > > > > > How would I customize endpoints with e.g. @Gzip annotations? > Would > > I at > > > the end still have my JAX-RS annotates resource classes? > > > > > > > > > These patches introduce a new "metamodel" concept, and move > > the current > > > specification of the RESTAPI based on XML schema and JAX-RS > > interfaces > > > to a new "model" built on the new metamodel. > > > > > > > > > What does this mean for you in practical terms? > Currently when > > you want > > > to introduce or modify one of the data types used by the > > RESTAPI you > > > start by modifying the XML schema. Once the patches are > merged > > the XML > > > schema will never be touched, as it will be automatically > > generated from > > > the "model". For example, imagine that you need to add a new > > "color" > > > attribute to the "VM" entity. To do so with the new > model you > > will have > > > to modify the following file, which is the specification of > > the "Vm" > > > entity, written as a Java interface: > > > > > > > > > > > > https://gerrit.ovirt.org/#/c/46478/16/backend/manager/modules/restapi/model/... > > > > > > In that interface you will have to add a line like this: > > > > > > String color(); > > > > > > Note that this Java interface is just the specification > of the > > entity, > > > it won't be used at all during runtime. Instead of that the > > XML schema > > > will be generated from it, and then Java will be generated > > from the XML > > > schema, as we do today (this will change in the future, but > > not yet). > > > > > > Same for the services. If you want to add a new "paint" > action > > to the > > > "Vm" resource then you won't modify the JAX-RS interfaces, > > instead of > > > that you will modify the following file, which is the > > specification of > > > the "Vm" service, written as a Java interface: > > > > > > > > > > > > https://gerrit.ovirt.org/#/c/47337/6/backend/manager/modules/restapi/model/s... > > > > > > In that interface you will need to add a > > > sub-interface > > representing the > > > action: > > > > > > interface Paint { > > > } > > > > > > The JAX-RS interface will be generated from that. > Currently these > > > sub-interfaces are empty. In the future they will > contain the > > > specifications of the parameters (currently in the > > rsdl_metadata.yml > > > file). > > > > > > > > > > > > These changes will currently affect only the > specification of the > > > RESTAPI, not the implementation, so in in the > > "Backend*Resource" classes > > > things won't change yet. > > > > > > > > > Currently I do not really understand where we are going > here. Are we > > > trying to get rid of rdsl? > > > > > > So basically two questions: > > > > > > 1) What is the final goal? > > > 2) What speaks agains using Hibernate validator on Daos > > > in > combination > > > with JAX-RS annotated resources (and just removing all > interfaces, as > > > far as I can see we only have one implementation per > endpoint) and > > > creating all schemas and clients through SWAGGER tooling? > > > > > > > > > If you have doubts, please let me know. > > > > > > Regards, > > > Juan Hernandez > > > > > > -- > > > Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. > Gorbea 3, > > planta > > > 3ºD, 28016 Madrid, Spain > > > Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - > > Red Hat > > > S.L. > > > _______________________________________________ > > > Devel mailing list > > > Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>> > > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>>> > > > http://lists.ovirt.org/mailman/listinfo/devel > > > > > > > > > Thanks, > > > > > > Roman > > > > > > -- > > Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, > planta > > 3ºD, 28016 Madrid, Spain > > Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 > > - > Red Hat > > S.L. > > > > > > > > I don't know if it is the right thing to do to invent something new > > here. I personally would prefer to thread a path which is very > common on > > the java community. > > I would love follow the DRY principle regarding to the stack and the > > code and would just use the great community projects there. > > > > It would also completely eliminate any custom magic. The JAX-RS > and CDI > > magic is pretty standard and easy to understand. > > From my perspective, real JAX-RS resoures have the advantage of > > > > * being very easy to understand (there is magic, but the > connection to > > the real endpoint is pretty clear) > > * being easy to customize suff, like adding @GZip to an annotation > > * describing pretty clearly the connection between the generated rest > > interface and the internal services > > > > Finally writing hand crafted tests is also much easier. > > > > What are your thoughts about that? > > > > Best Regards, > > Roman > > > > > > [1] https://github.com/rmohr/jetty-maven-cdi-demo > > [2] > > > https://github.com/rmohr/jetty-maven-cdi-demo/blob/master/src/main/java/rmoh... > > [3] http://swagger.io/ > > [4] https://github.com/kongchen/swagger-maven-plugin > > [5] https://github.com/swagger-api/swagger-codegen > > [6] > > > https://github.com/rmohr/jetty-maven-cdi-demo/blob/master/src/main/java/rmoh... > > [7] > > > http://maxenglander.com/2013/04/23/basic-restful-api-versioning-in-jersey.ht... > > [8] https://github.com/swagger-api/swagger-ui > > > > [9] > https://github.com/swagger-api/swagger-core/blob/master/modules/swagger-jaxr... >

-- Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta 3ºD, 28016 Madrid, Spain Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L. _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel

_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel

-- Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta 3ºD, 28016 Madrid, Spain Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.

On 10/27/2015 02:10 PM, Roman Mohr wrote:

On Tue, Oct 27, 2015 at 1:45 PM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com>> wrote:

On 10/27/2015 12:55 PM, Roman Mohr wrote: > > > On Tue, Oct 27, 2015 at 12:16 PM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> wrote: > > On 10/27/2015 11:28 AM, Roman Mohr wrote: > > > > > > On Tue, Oct 27, 2015 at 10:47 AM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>> wrote: > > > > On 10/27/2015 10:16 AM, Roman Mohr wrote: > > > > > > > > > On Mon, Oct 26, 2015 at 5:32 PM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> > > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>>> wrote: > > > > > > On 10/26/2015 04:56 PM, Roman Mohr wrote: > > > > Hi Juan, > > > > > > > > The way to specify the contract look pretty clean and > nice. > > > > I would love to read a few words about the big > picture. What > > is the > > > > final scenario? > > > > > > > > > > The motivation for this change is that currently we > don't have > > a central > > > place where the RESTAPI is specified, rather we have several > > different > > > places, using several different technologies: > > > > > > * XML schema for the data model. > > > * JAX-RS for part of the operational model (without the > > parameters). > > > * rsdl_metadata.yaml for the parameters of the > operational model. > > > > > > This makes it difficult to infer information about the > model. For > > > example, the generators of the SDKs have to download the XML > > schema, and > > > the RSDL (which is generated from the JAX-RS interfaces > using > > reflection > > > and combining it with the information from the > > rsdl_metadata.yaml file) > > > and then they have to do their own computations to extract > > what they > > > need. > > > > > > Same happens with the CLI: it has to extract the information > > it needs > > > from the Python code generated for the Python SDK, yet > another > > level of > > > indirection. > > > > > > > > > You are right, that definitely needs to be cleaned up. I > just want to > > > discuss a few points below with you. > > > > > > > > > > > > We are also lacking a comprehensive reference > documentation of the > > > RESTAPI. What we currently have has been written by > hand, and > > gets out > > > of sync very quickly, and we don't even notice. > > > > > > > > > Did you also consider swagger? It is made for exactly that > purpose. > > > I created a demo in [1] which uses resteasy, weld, > hibernate-validator > > > and swagger to demonstrate how to do DRY with jaxrs. > > > Would be great to hear you thoughts on that. > > > > > > And there is the great swagger-ui [8] to display the > documentation > > in a > > > more human readable way. > > > > > > > Yes, I considered Swagger, and rejected it because it is JSON > centric, > > and I think JSON isn't as good as Java to represent the > contracts of our > > RESTAPI. > > > > > > You just write plain jax-rs, swagger just creates a description out of > > it. So the source defining the contract is pure java (jax-rs with > some > > swagger annotations for description, etc.). > > Or am I missing the point here? > > > > If I understand correctly the Swagger core is a JSON (or YAML) > specification of the API. From that you can generate JAX-RS annotated > code, not the other way around. So the specification document that you > write is a JSON document. > > > You are right, my terminology here was not clear. Swagger is just a > specification. Swagger-core and swagger-jaxrs are the ones which can > create the documnetation out of JAX-RS resources. > > > Alternatively, you can use the Swagger annotations to decorate your > implementation, both the entity classes and the JAX-RS resource > implementations, and then extract the model from that. But this is > putting the implementation before the specification. That is where we > are today, and it causes multiple problems. I think it is better to have > the specification and the implementation separate. Swagger does that > well when using JSON directly, our metamodel also does it well, but > using a better language. > > > > Isn't our problem that we have everything scattered arount the place and > not that we are using JAX-RS? I don't think that this has anything to do > with specification before implementation or implementation before > specification. >

Correct, the problem with the specification if that we don't have one, just pieces of it scattered around, some in the XML schema, some in the JAX-RS interfaces, some in the rsdl_metadata.yaml file.

This is historically related with the order of specification and implementation. When the project started we didn't create a specification of the RESTAPI, only an implementation, using JAX-RS and XML schema. Then we added the rsdl_metadata.yaml file, and extracted a specification from that, the RSDL. The result is hard to consume. To make it easy to consume we need to reverse the order: first specification, free of implementation details, then implementation.

> > > > > > > In addition we need to do these changes in a smooth way, without causing > > big changes in the middle. For example, in the first step we need to > > preserve the JAX-RS interfaces as they are today, to avoid massive > > changes to all the resource implementations. This could be done with > > > > Swagger, but would require custom code generators. With less effort we > > can do our own. > > > > > > This is of course generally a difficult task. But I do not know why it > > would be more difficult to write a custom swagger reader (if we even > > have to, it can read the interfaces as well) . > > They are pretty streight forward. Just look at [9], this contains the > > wole jax-rs specific code to generate the swagger documentation. > > > > But yes, I don't know every detail here of the engine and can't clearly > > say that integrating that would just streight forward (my feeling tells > > me that it would not be too hard). I am just under the impression that > > we would benefit from that. Just reduces custom magic to a minimum. > > > > Using something like Swagger would be certainly possible, and not that > hard, but it requires an effort. For example, say that we decide to use > the Swagger annotations. Then we will need to add these annotations to > all our JAX-RS resource implementations. That is a non trivial effort. > We would need to add the annotations to the entities as well. But wait, > we don't have such entities, only XML schema. So we would need a reader > that parses XML schema, and creating it requires effort. > > > You can just create the entities/daos once like we do now on every build > and annotate them once and drop the whole xml. >

The DAOs aren't part of the RESTAPI. The backend entities aren't either. The RESTAPI has its own entities which are currently generated from XML schema.

s/DAOs/DTOs I meant Data transfer object, not Data Access objects. And they are generated.

> > Where do we put > the documentation then? Part in the JAX-RS interfaces, part in the XML > schema. > > > I don't understand that. There is only one documentation and > implementation source, that is the JAX-RS resource and the entity/dao > accepted by the endpoint. > The XML schema can just be removed. The documentation in swagger format > for other tools like swagger-codegen and swagger-ci can be made > available through swagger-maven-plugin or a servlet in the engine which > generates the swagger json on the fly (like I did in [1]) >

Those are two: JAX-RS and entities, and they are implementations, not specifications.

The XML schema can't be drop, as many clients use it. But we can generate it from the model, and that is what we will do.

Did not look into that.

> > We are already there, and we ended up with no documentation at > all. In my view the sum of these efforts is higher than doing our own > metamodel. > > > > > > > > > > > Swagger UI is certainly great. I did test it and it is really > good. We > > may be able to copy some concepts. > > > > > > > > > > > To solve these issues I intend to have the specification of > > the RESTAPI > > > only in one place, and using only one technology. I > decided to > > use Java > > > interfaces for that. Note however that they are just the > > support for the > > > information, like paper is the support for ink. I decided to > > use Java > > > because it is easy to create, modify and re-factor using > tools > > familiar > > > to most of us. > > > > > > These source of these interfaces is analysed (using QDox, > > currently) and > > > a "model" of the RESTAPI is generated in memory. This > model is > > > independent of the supporting Java source, and easy to > > consume. For > > > example, imagine that you want to list all the types > available > > in the > > > model and for each one display its documentation: > > > > > > Model model = ...; > > > for (Type type : model.getTypes()) { > > > Name name = type.getName(); > > > String doc = type.getDoc(); > > > System.out.println(name + ": " + doc); > > > } > > > > > > Something like this, but more elaborate, will be part of > a web > > > application that provides comprehensive reference > documentation, > > > assuming that we dedicate the time to write documentation > > comments in > > > the specification. > > > > > > I intend to use this model also to do simplify the > generators > > of the > > > SDKs and the CLI. > > > > > > In addition these are some of the things that I would > like to > > change in > > > the near future (for 4.0): > > > > > > * Move the specification of the parameters of operations out > > of the > > > rsdl_metadata.yaml file and into the model. For example: > > > > > > @Service > > > public VmService { > > > /** > > > * The operation to add a virtual machine. > > > */ > > > interface Add { > > > /** > > > * The representation of the virtual machine is > received > > > * as parameter, and the representation of the created > > > * virtual machine is returned as result. > > > */ > > > @In @Out Vm vm(); > > > > > > /** > > > * In the future, we will be able to specify other > > > * parameters here. > > > */ > > > @In Boolean force(); > > > > > > /** > > > * Even with default values. > > > */ > > > @In default Boolean force() { return true; } > > > > > > /** > > > * And we will be able to specify constraints, which > > > * will replace the rsdl_metadata.yaml file. > > > */ > > > @Constraint > > > default boolean vmNameMustNotBeNull() { > > > return vm().name() != null; > > > } > > > } > > > } > > > > > > * Enforce the constraints automatically. If the constraints > > are in the > > > model, then we can just check them and reject requests > before > > delivering > > > them to the application. Currently we do this manually > (and often > > > forget) with calls to "validate(...)" methods. > > > > > > > > > > > > Did you consider just annotating the DTOs with JSR-303 > annotations and > > > integrate a validator with jax-rs? > > > See [2] for an example. > > > > > > > This is a great way to implement a system, but the goal here > isn't to > > implement it, rather to specify it. Using annotations in this > way won't > > help the generators of the SDKs, for example, to figure out what > > parameters are required, mandatory, etc. > > > > > > Swagger understands them. From my example project, swagger created > that > > > > description: > > type: "string" > > minLength: 10 > > maxLength: 100 > > > > out of > > > > @Size(min=10, max=100) # jsr-303 > > private String description; > > > > and so does swagger-codegen which can generate clients in java, > python, ... > > > > This is extracting the specification from the implementation, which > isn't correct in my opinion, it should be the opposite. Not saying that > this makes Swagger bad, it is nice that it has this capability, but I > think we can do it better. > > > In my opition this is the main advantage of that. It is DRY while still > having full control of the implementation. >

I think that DRY is better served if you write the specification once, and then, from that, you generate the contracts (interfaces, entities, builders, etc) that the implementation should use.

And how do you add special things like @Gzip or other specializations which you might use?

There shouldn't be any specialization. All the RESTAPI services should behave exactly the same. So if one of them supports GZIP, for example, then all of the should do. Anyhow, if there is a real need for a specialization it can go into the implementation, it isn't a problem.

> > > > > > > > > > * Generate the Java classes directly from the model. Instead of Model -> > > > XML Schema -> Java, we can do Model -> Java. This will allow us to solve > > > some of the XJC compiler limitations, like the horrible way we handle > > > arrays today. > > > > > > > > > Swagger [3] is a rest documentation specification. There is also a maven > > > plugin [4] and you can create clients for example with [5]. > > > > > > > > > > > > * Replace JAX-RS with a simpler infrastructure that supports better > > > streaming and CDI injection. > > > > > > > > > > > > With resteasy-cdi you have pretty good injection support for resteasy. > > > Run the demo in [1] to see it in action and look at the file at [6]. > > > > > > > Resteasy-CDI isn't standard, it only works with Resteasy. If we rely on > > it then we re tied to Resteasy for ever. > > > > > > Even jersey has support for that (I think it is called jeryse-gf-cdi), > > but why would we want switch? I don't think that jboss will drop > > resteasy and it also works fine outside of full blown containers. I > > don't think that this is an argument. > > > > Well, nobody thought that JBoss would drop Tomcat, and they did. Nobody > thought that Resteasy would change the SPI from 2.x to 3.x, and they > did.\ > > > It will be there for the next year or another library which offers the > same thing. Having injections in Jax-rs resources is important for > spring, jboss, glassfish and others there will always be ways to do > that. I don't know why we should create our own 'custom standard' just > because another standard does not include dependency injection when we > can just extend it so easily. >

Here our views of "easily" are different. I have suffered the problems with changes in Resteasy over the past years, and I won't describe it as easy.

I agree. Maven and dependencies are never easy. But I prefer to fix something now and then on regular maven dependency updates than being tied to somesing because of a custom code generator.

> > We want (well, I want) to get out of JAX-RS because it doesn't support > well CDI and streaming. > > > Why should streaming resources be in the rest interface? >

Streaming should be used not only in the RESTAPI, but in the complete system, including the backend and the DAL, otherwise we have serious problems when clients requests large amounts of objects.

Streaming is no solution here. It does not address the problem at all. Solving the database bottleneck with this approach is just wrong. A client can iterate over our 10000 VMs extremely fast. No matter if streamed or not, so this will still hit the database.

Streaming is a mechanism to improve the latency when serving multiple objects: instead of having your client waiting till you have all the requested objects (10 or 10K) in memory you send them as soon as you can. In addition it also reduces the use of memory in the server. Think about watching a movie. What is better, downloading it complete and the watch it, or watch as the individual frames arrive?

What you normally do to prevent this:

1) A second level cache (see here for a start https://gerrit.ovirt.org/#/q/status:open+project:ovirt-engine+branch:master+...)

Second level cache doesn't solve the latency problem. If someone requests 1000 VMs he will still have to wait till you have the 1000 VMs arranged in memory before sending them.

2) Add paging to the rest interface and define a maximum number per page (would you trottle the stream in your approach per session, request?)

The client then just happily fetch one page after the other without thinking about overloading your system (but of course the client can still deliberately harm you), because a client can assume that you optimize your application (caches, daos and database and page size per request) for that.

The RESTAPI already has a pagination mechanism, but some clients don't want to use it. We used to have a hard limit and we had to remove it.

Of course for notifications streams are great, but again why should they be in the rest interface? Adding websockets or other stuff to listen on changed VMs would be great. Just discussed this recently with rgolan.

Notifications are currently out of the scope of the RESTAPI, and there are no plans to add them.

Think, for example, that a client requests 100 virtual machines. What we currently do is the following:

* The DAL requests the 100 rows to the database, and waits till the 100 rows are loaded in memory (this is how the JDBC driver works by default). Only when the 100 rows are in memory the are converted to backend objects.

* The DAL returns the 100 backend objects to the BLL, which processes them. Only when they are all processed they are returned to the RESTAPI, as a list.

* The RESTAPI transforms the 100 backend objects into RESTAPI objects. Only when the are all transformed they are returned by the corresponding JAX-RS method to the JAX-RS infrastructure. This is encouraged by the high level interface of JAX-RS, as it encourages you to write methods like this:

@GET Vms get()

Who could ever have thought that this can be a good idea. :D I would go with paging instead of streams for that purpose (just make sure to sort the returned values repeatable).

This is just what JAX-RS encourages.

* Once the JAX-RS infrastructure has the complete result it starts to convert it to the XML (or JSON) document.

Same for the reverse direction.

Put 10K virtual machines instead of just 100 and you see the problem.

Of course one could use JAX-RS in a way that doesn't have this problem, it is certainly possible, you can get the output stream and serialize the result from the resource implementation, in a an streaming fashion. But then the value of JAX-RS is reduced, you can do the same with a simpler servlet architecture. That is what I plan to do.

Exactly my point. I would do monitoring stuff with different simple stream servlets.

Yes, servlet technology is enough and better for that, and for serving the RESTAPI content as well.

> > > > > > > > > > > > * Add support for multiple versions of the API, using > the "Version" > > > header, and generating different Java classes for > entities and services. > > > For example, if we have versions 4 and 5 of the model as > separate > > > artifacts, then we can generate "V4Vm" and "V5Vm" entity > classes, and > > > "V4VmService" and "V5VmService" service classes. These > can be used > > > simultaneously in the server, so we can have in the same > engine > > > implementations for multiple versions. > > > > > > > > > There are also many ways to do that. Here [7] is a pretty > clean way to > > > do it with jax-rs and you will have everything related in > one resource. > > > > > > > Yes, there are many ways. In my opinion it is better to use > the HTTP > > "Version" header, and to forward requests to different resource > > implementations without requiring different URLs or different > > content types. > > > > Have no strong opinion there, just seemed to be a good choice > regarding > > to versioning limitations in jax-rs and our use of jax-rs > subresources. > > > > > > > > > > > > > The final picture isn't completely defined yet. > > > > > > Regards, > > > Juan Hernandez > > > > > > > On Mon, Oct 26, 2015 at 4:03 PM, Juan Hernández > <jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>> > > > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>>>> wrote: > > > > > > > > Hello, > > > > > > > > I will soon merge the following patches that > introduce a new > > > way to > > > > specify the contracts of the RESTAPI: > > > > > > > > restapi: Introduce metamodel > > > > https://gerrit.ovirt.org/45852 > > > > > > > > restapi: Use metamodel > > > > https://gerrit.ovirt.org/46478 > > > > > > > > restapi: Generate JAX-RS interfaces from model > > > > https://gerrit.ovirt.org/47337 > > > > > > > > > > > > > > > Looks pretty much like we are replacing one way of > > annotating things > > > > with another way of specifying things. > > > > Could you elaborate what the benefit of that way of > > description is? > > > > > > > > How would I customize endpoints with e.g. @Gzip > annotations? > > Would > > > I at > > > > the end still have my JAX-RS annotates resource classes? > > > > > > > > > > > > These patches introduce a new "metamodel" concept, > and move > > > the current > > > > specification of the RESTAPI based on XML schema > and JAX-RS > > > interfaces > > > > to a new "model" built on the new metamodel. > > > > > > > > > > > > What does this mean for you in practical terms? > > Currently when > > > you want > > > > to introduce or modify one of the data types used > by the > > > RESTAPI you > > > > start by modifying the XML schema. Once the > patches are > > merged > > > the XML > > > > schema will never be touched, as it will be > automatically > > > generated from > > > > the "model". For example, imagine that you need to > add a new > > > "color" > > > > attribute to the "VM" entity. To do so with the new > > model you > > > will have > > > > to modify the following file, which is the > specification of > > > the "Vm" > > > > entity, written as a Java interface: > > > > > > > > > > > > > > > > > > https://gerrit.ovirt.org/#/c/46478/16/backend/manager/modules/restapi/model/... > > > > > > > > In that interface you will have to add a line like > this: > > > > > > > > String color(); > > > > > > > > Note that this Java interface is just the > specification > > of the > > > entity, > > > > it won't be used at all during runtime. Instead of > that the > > > XML schema > > > > will be generated from it, and then Java will be > generated > > > from the XML > > > > schema, as we do today (this will change in the > future, but > > > not yet). > > > > > > > > Same for the services. If you want to add a new > "paint" > > action > > > to the > > > > "Vm" resource then you won't modify the JAX-RS > interfaces, > > > instead of > > > > that you will modify the following file, which is the > > > specification of > > > > the "Vm" service, written as a Java interface: > > > > > > > > > > > > > > > > > > https://gerrit.ovirt.org/#/c/47337/6/backend/manager/modules/restapi/model/s... > > > > > > > > In that interface you will need to add a sub-interface > > > representing the > > > > action: > > > > > > > > interface Paint { > > > > } > > > > > > > > The JAX-RS interface will be generated from that. > > Currently these > > > > sub-interfaces are empty. In the future they will > > contain the > > > > specifications of the parameters (currently in the > > > rsdl_metadata.yml > > > > file). > > > > > > > > > > > > > > > > These changes will currently affect only the > > specification of the > > > > RESTAPI, not the implementation, so in in the > > > "Backend*Resource" classes > > > > things won't change yet. > > > > > > > > > > > > Currently I do not really understand where we are going > > here. Are we > > > > trying to get rid of rdsl? > > > > > > > > So basically two questions: > > > > > > > > 1) What is the final goal? > > > > 2) What speaks agains using Hibernate validator on Daos in > > combination > > > > with JAX-RS annotated resources (and just removing all > > interfaces, as > > > > far as I can see we only have one implementation per > > endpoint) and > > > > creating all schemas and clients through SWAGGER tooling? > > > > > > > > > > > > If you have doubts, please let me know. > > > > > > > > Regards, > > > > Juan Hernandez > > > > > > > > -- > > > > Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. > > Gorbea 3, > > > planta > > > > 3ºD, 28016 Madrid, Spain > > > > Inscrita en el Reg. Mercantil de Madrid – C.I.F. > B82657941 - > > > Red Hat > > > > S.L. > > > > _______________________________________________ > > > > Devel mailing list > > > > Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>> > > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>>> > > > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>> > > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>>>> > > > > http://lists.ovirt.org/mailman/listinfo/devel > > > > > > > > > > > > Thanks, > > > > > > > > Roman > > > > > > > > > -- > > > Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. > Gorbea 3, > > planta > > > 3ºD, 28016 Madrid, Spain > > > Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - > > Red Hat > > > S.L. > > > > > > > > > > > > I don't know if it is the right thing to do to invent > something new > > > here. I personally would prefer to thread a path which is very > > common on > > > the java community. > > > I would love follow the DRY principle regarding to the stack > and the > > > code and would just use the great community projects there. > > > > > > It would also completely eliminate any custom magic. The JAX-RS > > and CDI > > > magic is pretty standard and easy to understand. > > > From my perspective, real JAX-RS resoures have the advantage of > > > > > > * being very easy to understand (there is magic, but the > > connection to > > > the real endpoint is pretty clear) > > > * being easy to customize suff, like adding @GZip to an > annotation > > > * describing pretty clearly the connection between the > generated rest > > > interface and the internal services > > > > > > Finally writing hand crafted tests is also much easier. > > > > > > What are your thoughts about that? > > > > > > Best Regards, > > > Roman > > > > > > > > > [1] https://github.com/rmohr/jetty-maven-cdi-demo > > > [2] > > > > > > https://github.com/rmohr/jetty-maven-cdi-demo/blob/master/src/main/java/rmoh... > > > [3] http://swagger.io/ > > > [4] https://github.com/kongchen/swagger-maven-plugin > > > [5] https://github.com/swagger-api/swagger-codegen > > > [6] > > > > > > https://github.com/rmohr/jetty-maven-cdi-demo/blob/master/src/main/java/rmoh... > > > [7] > > > > > > http://maxenglander.com/2013/04/23/basic-restful-api-versioning-in-jersey.ht... > > > [8] https://github.com/swagger-api/swagger-ui > > > > > > > [9] > > > https://github.com/swagger-api/swagger-core/blob/master/modules/swagger-jaxr... > > >

-- Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta 3ºD, 28016 Madrid, Spain Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.

On 10/27/2015 03:05 PM, Roman Mohr wrote:

On Tue, Oct 27, 2015 at 2:41 PM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com>> wrote:

On 10/27/2015 02:10 PM, Roman Mohr wrote: > > > On Tue, Oct 27, 2015 at 1:45 PM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> wrote: > > On 10/27/2015 12:55 PM, Roman Mohr wrote: > > > > > > On Tue, Oct 27, 2015 at 12:16 PM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>> wrote: > > > > On 10/27/2015 11:28 AM, Roman Mohr wrote: > > > > > > > > > On Tue, Oct 27, 2015 at 10:47 AM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> > > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>>> wrote: > > > > > > On 10/27/2015 10:16 AM, Roman Mohr wrote: > > > > > > > > > > > > On Mon, Oct 26, 2015 at 5:32 PM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>> > > > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>>>> wrote: > > > > > > > > On 10/26/2015 04:56 PM, Roman Mohr wrote: > > > > > Hi Juan, > > > > > > > > > > The way to specify the contract look pretty > clean and > > nice. > > > > > I would love to read a few words about the big > > picture. What > > > is the > > > > > final scenario? > > > > > > > > > > > > > The motivation for this change is that currently we > > don't have > > > a central > > > > place where the RESTAPI is specified, rather we > have several > > > different > > > > places, using several different technologies: > > > > > > > > * XML schema for the data model. > > > > * JAX-RS for part of the operational model > (without the > > > parameters). > > > > * rsdl_metadata.yaml for the parameters of the > > operational model. > > > > > > > > This makes it difficult to infer information about the > > model. For > > > > example, the generators of the SDKs have to > download the XML > > > schema, and > > > > the RSDL (which is generated from the JAX-RS > interfaces > > using > > > reflection > > > > and combining it with the information from the > > > rsdl_metadata.yaml file) > > > > and then they have to do their own computations to > extract > > > what they > > > > need. > > > > > > > > Same happens with the CLI: it has to extract the > information > > > it needs > > > > from the Python code generated for the Python SDK, yet > > another > > > level of > > > > indirection. > > > > > > > > > > > > You are right, that definitely needs to be cleaned up. I > > just want to > > > > discuss a few points below with you. > > > > > > > > > > > > > > > > We are also lacking a comprehensive reference > > documentation of the > > > > RESTAPI. What we currently have has been written by > > hand, and > > > gets out > > > > of sync very quickly, and we don't even notice. > > > > > > > > > > > > Did you also consider swagger? It is made for exactly that > > purpose. > > > > I created a demo in [1] which uses resteasy, weld, > > hibernate-validator > > > > and swagger to demonstrate how to do DRY with jaxrs. > > > > Would be great to hear you thoughts on that. > > > > > > > > And there is the great swagger-ui [8] to display the > > documentation > > > in a > > > > more human readable way. > > > > > > > > > > Yes, I considered Swagger, and rejected it because it is > JSON > > centric, > > > and I think JSON isn't as good as Java to represent the > > contracts of our > > > RESTAPI. > > > > > > > > > You just write plain jax-rs, swagger just creates a > description out of > > > it. So the source defining the contract is pure java > (jax-rs with > > some > > > swagger annotations for description, etc.). > > > Or am I missing the point here? > > > > > > > If I understand correctly the Swagger core is a JSON (or YAML) > > specification of the API. From that you can generate JAX-RS > annotated > > code, not the other way around. So the specification document > that you > > write is a JSON document. > > > > > > You are right, my terminology here was not clear. Swagger is just a > > specification. Swagger-core and swagger-jaxrs are the ones which can > > create the documnetation out of JAX-RS resources. > > > > > > Alternatively, you can use the Swagger annotations to decorate > your > > implementation, both the entity classes and the JAX-RS resource > > implementations, and then extract the model from that. But this is > > putting the implementation before the specification. That is > where we > > are today, and it causes multiple problems. I think it is > better to have > > the specification and the implementation separate. Swagger > does that > > well when using JSON directly, our metamodel also does it > well, but > > using a better language. > > > > > > > > Isn't our problem that we have everything scattered arount the > place and > > not that we are using JAX-RS? I don't think that this has anything > to do > > with specification before implementation or implementation before > > specification. > > > > Correct, the problem with the specification if that we don't have one, > just pieces of it scattered around, some in the XML schema, some in the > JAX-RS interfaces, some in the rsdl_metadata.yaml file. > > This is historically related with the order of specification and > implementation. When the project started we didn't create a > specification of the RESTAPI, only an implementation, using JAX-RS and > XML schema. Then we added the rsdl_metadata.yaml file, and extracted a > specification from that, the RSDL. The result is hard to consume. To > make it easy to consume we need to reverse the order: first > specification, free of implementation details, then implementation. > > > > > > > > > > > > In addition we need to do these changes in a smooth way, without causing > > > big changes in the middle. For example, in the first step we need to > > > preserve the JAX-RS interfaces as they are today, to avoid massive > > > changes to all the resource implementations. This could be done with > > > > > > Swagger, but would require custom code generators. With less effort we > > > can do our own. > > > > > > > > > This is of course generally a difficult task. But I do not know why it > > > would be more difficult to write a custom swagger reader (if we even > > > have to, it can read the interfaces as well) . > > > They are pretty streight forward. Just look at [9], this contains the > > > wole jax-rs specific code to generate the swagger documentation. > > > > > > But yes, I don't know every detail here of the engine and can't clearly > > > say that integrating that would just streight forward (my feeling tells > > > me that it would not be too hard). I am just under the impression that > > > we would benefit from that. Just reduces custom magic to a minimum. > > > > > > > Using something like Swagger would be certainly possible, and not that > > hard, but it requires an effort. For example, say that we decide to use > > the Swagger annotations. Then we will need to add these annotations to > > all our JAX-RS resource implementations. That is a non trivial effort. > > We would need to add the annotations to the entities as well. But wait, > > we don't have such entities, only XML schema. So we would need a reader > > that parses XML schema, and creating it requires effort. > > > > > > You can just create the entities/daos once like we do now on every build > > and annotate them once and drop the whole xml. > > > > The DAOs aren't part of the RESTAPI. The backend entities aren't either. > The RESTAPI has its own entities which are currently generated from XML > schema. > > > s/DAOs/DTOs I meant Data transfer object, not Data Access objects. And > they are generated. > > > > > > > Where do we put > > the documentation then? Part in the JAX-RS interfaces, part in the XML > > schema. > > > > > > I don't understand that. There is only one documentation and > > implementation source, that is the JAX-RS resource and the entity/dao > > accepted by the endpoint. > > The XML schema can just be removed. The documentation in swagger format > > for other tools like swagger-codegen and swagger-ci can be made > > available through swagger-maven-plugin or a servlet in the engine which > > generates the swagger json on the fly (like I did in [1]) > > > > Those are two: JAX-RS and entities, and they are implementations, not > specifications. > > The XML schema can't be drop, as many clients use it. But we can > generate it from the model, and that is what we will do. > > > Did not look into that. > > > > > > > We are already there, and we ended up with no documentation at > > all. In my view the sum of these efforts is higher than doing > our own > > metamodel. > > > > > > > > > > > > > > > > > > > Swagger UI is certainly great. I did test it and it is > really > > good. We > > > may be able to copy some concepts. > > > > > > > > > > > > > > > To solve these issues I intend to have the > specification of > > > the RESTAPI > > > > only in one place, and using only one technology. I > > decided to > > > use Java > > > > interfaces for that. Note however that they are > just the > > > support for the > > > > information, like paper is the support for ink. I > decided to > > > use Java > > > > because it is easy to create, modify and re-factor > using > > tools > > > familiar > > > > to most of us. > > > > > > > > These source of these interfaces is analysed > (using QDox, > > > currently) and > > > > a "model" of the RESTAPI is generated in memory. This > > model is > > > > independent of the supporting Java source, and easy to > > > consume. For > > > > example, imagine that you want to list all the types > > available > > > in the > > > > model and for each one display its documentation: > > > > > > > > Model model = ...; > > > > for (Type type : model.getTypes()) { > > > > Name name = type.getName(); > > > > String doc = type.getDoc(); > > > > System.out.println(name + ": " + doc); > > > > } > > > > > > > > Something like this, but more elaborate, will be > part of > > a web > > > > application that provides comprehensive reference > > documentation, > > > > assuming that we dedicate the time to write > documentation > > > comments in > > > > the specification. > > > > > > > > I intend to use this model also to do simplify the > > generators > > > of the > > > > SDKs and the CLI. > > > > > > > > In addition these are some of the things that I would > > like to > > > change in > > > > the near future (for 4.0): > > > > > > > > * Move the specification of the parameters of > operations out > > > of the > > > > rsdl_metadata.yaml file and into the model. For > example: > > > > > > > > @Service > > > > public VmService { > > > > /** > > > > * The operation to add a virtual machine. > > > > */ > > > > interface Add { > > > > /** > > > > * The representation of the virtual machine is > > received > > > > * as parameter, and the representation of > the created > > > > * virtual machine is returned as result. > > > > */ > > > > @In @Out Vm vm(); > > > > > > > > /** > > > > * In the future, we will be able to > specify other > > > > * parameters here. > > > > */ > > > > @In Boolean force(); > > > > > > > > /** > > > > * Even with default values. > > > > */ > > > > @In default Boolean force() { return true; } > > > > > > > > /** > > > > * And we will be able to specify > constraints, which > > > > * will replace the rsdl_metadata.yaml file. > > > > */ > > > > @Constraint > > > > default boolean vmNameMustNotBeNull() { > > > > return vm().name() != null; > > > > } > > > > } > > > > } > > > > > > > > * Enforce the constraints automatically. If the > constraints > > > are in the > > > > model, then we can just check them and reject requests > > before > > > delivering > > > > them to the application. Currently we do this manually > > (and often > > > > forget) with calls to "validate(...)" methods. > > > > > > > > > > > > > > > > Did you consider just annotating the DTOs with JSR-303 > > annotations and > > > > integrate a validator with jax-rs? > > > > See [2] for an example. > > > > > > > > > > This is a great way to implement a system, but the goal here > > isn't to > > > implement it, rather to specify it. Using annotations in > this > > way won't > > > help the generators of the SDKs, for example, to figure > out what > > > parameters are required, mandatory, etc. > > > > > > > > > Swagger understands them. From my example project, swagger > created > > that > > > > > > description: > > > type: "string" > > > minLength: 10 > > > maxLength: 100 > > > > > > out of > > > > > > @Size(min=10, max=100) # jsr-303 > > > private String description; > > > > > > and so does swagger-codegen which can generate clients in java, > > python, ... > > > > > > > This is extracting the specification from the implementation, > which > > isn't correct in my opinion, it should be the opposite. Not > saying that > > this makes Swagger bad, it is nice that it has this > capability, but I > > think we can do it better. > > > > > > In my opition this is the main advantage of that. It is DRY while > still > > having full control of the implementation. > > > > I think that DRY is better served if you write the specification once, > and then, from that, you generate the contracts (interfaces, entities, > builders, etc) that the implementation should use. > > > And how do you add special things like @Gzip or other specializations > which you might use? >

There shouldn't be any specialization. All the RESTAPI services should behave exactly the same. So if one of them supports GZIP, for example, then all of the should do.

Anyhow, if there is a real need for a specialization it can go into the implementation, it isn't a problem.

> > > > > > > > > > > > > > > > * Generate the Java classes directly from the > model. Instead of Model -> > > > > XML Schema -> Java, we can do Model -> Java. This > will allow us to solve > > > > some of the XJC compiler limitations, like the > horrible way we handle > > > > arrays today. > > > > > > > > > > > > Swagger [3] is a rest documentation specification. > There is also a maven > > > > plugin [4] and you can create clients for example with > [5]. > > > > > > > > > > > > > > > > * Replace JAX-RS with a simpler infrastructure > that supports better > > > > streaming and CDI injection. > > > > > > > > > > > > > > > > With resteasy-cdi you have pretty good injection > support for resteasy. > > > > Run the demo in [1] to see it in action and look at > the file at [6]. > > > > > > > > > > Resteasy-CDI isn't standard, it only works with > Resteasy. If we rely on > > > it then we re tied to Resteasy for ever. > > > > > > > > > Even jersey has support for that (I think it is called > jeryse-gf-cdi), > > > but why would we want switch? I don't think that jboss will drop > > > resteasy and it also works fine outside of full blown > containers. I > > > don't think that this is an argument. > > > > > > > Well, nobody thought that JBoss would drop Tomcat, and they > did. Nobody > > thought that Resteasy would change the SPI from 2.x to 3.x, > and they > > did.\ > > > > > > It will be there for the next year or another library which offers the > > same thing. Having injections in Jax-rs resources is important for > > spring, jboss, glassfish and others there will always be ways to do > > that. I don't know why we should create our own 'custom standard' just > > because another standard does not include dependency injection when we > > can just extend it so easily. > > > > Here our views of "easily" are different. I have suffered the problems > with changes in Resteasy over the past years, and I won't describe it as > easy. > > > I agree. Maven and dependencies are never easy. But I prefer to fix > something now and then on regular maven dependency updates than being > tied to somesing because of a custom code generator. > > > > > > We want (well, I want) to get out of JAX-RS because it doesn't support > > well CDI and streaming. > > > > > > Why should streaming resources be in the rest interface? > > > > Streaming should be used not only in the RESTAPI, but in the complete > system, including the backend and the DAL, otherwise we have serious > problems when clients requests large amounts of objects. > > > Streaming is no solution here. It does not address the problem at all. > Solving the database bottleneck with this approach is just wrong. > A client can iterate over our 10000 VMs extremely fast. No matter if > streamed or not, so this will still hit the database. >

Streaming is a mechanism to improve the latency when serving multiple objects: instead of having your client waiting till you have all the requested objects (10 or 10K) in memory you send them as soon as you can.

1) you would still fetch everything in batches from the database to reduce the database load

Correct, the database would also work better if the DAL retrieved things using a streaming approach, but that is out of the scope of the RESTAPI.

2) the latency regarding to jax-rs is almost zero. We have bugs like https://bugzilla.redhat.com/show_bug.cgi?id=1268224. You would not see a human noticable improvement regarding to latency. Even if they would be fixed (well yes the first vms would drop out faster, but the over all execution time might even be worse, that depends on the underlying database fetches).

Indeed, issues like the one described in that bug also need to be fixed. Streaming results isn't the solution for all problems, just one ingredient of the overall solution, caching being another important one.

In addition it also reduces the use of memory in the server.

Which is absolutely not an issue for the engine. We can have the whole database in memory and we would still not need much memory for that. And that is why second level caches have eviction strategies.

We don't have a huge database, so we already have the whole database in memory, inside the database server caches. Why aren't we faster then? Is the latency of the communications between the engine and the database server related? Using less resources is almost always good, even if you have plenty of them. For example, it is good if you can return 1000 VMs using 1 KiB of RAM instead of 1 GiB: less CPU cache misses, less context switches, less work for the garbage collector, thus faster and less power hungry systems.

Think about watching a movie. What is better, downloading it complete and the watch it, or watch as the individual frames arrive?

> What you normally do to prevent this: > > 1) A second level cache (see here for a start > https://gerrit.ovirt.org/#/q/status:open+project:ovirt-engine+branch:master+...)

Second level cache doesn't solve the latency problem. If someone requests 1000 VMs he will still have to wait till you have the 1000 VMs arranged in memory before sending them.

You are right, it does not reduce serialzation latency, but latency because of the REST api serialization will never be a problem here. It will always be our implementation which is the problem. With and without streams.

Correct, if the RESTAPI has to wait doing nothing while other components prepare all the results then there is no value in streaming, that is why all the components should work in an streaming fashion.

> 2) Add paging to the rest interface and define a maximum number per > page (would you trottle the stream in your approach per session, request?) > > The client then just happily fetch one page after the other without > thinking about overloading your system (but of course the client can > still deliberately harm you), because a client can assume that you > optimize your application (caches, daos and database and page size per > request) for that. >

The RESTAPI already has a pagination mechanism, but some clients don't want to use it. We used to have a hard limit and we had to remove it.

I cant't argue with that. That is definitely an interesting fact.

> Of course for notifications streams are great, but again why should they > be in the rest interface? Adding websockets or other stuff to listen on > changed VMs would be great. Just discussed this recently with rgolan. >

Notifications are currently out of the scope of the RESTAPI, and there are no plans to add them.

> > Think, for example, that a client requests 100 virtual machines. What we > currently do is the following: > > * The DAL requests the 100 rows to the database, and waits till the 100 > rows are loaded in memory (this is how the JDBC driver works by > default). Only when the 100 rows are in memory the are converted to > backend objects. > > * The DAL returns the 100 backend objects to the BLL, which processes > them. Only when they are all processed they are returned to the RESTAPI, > as a list. > > * The RESTAPI transforms the 100 backend objects into RESTAPI objects. > Only when the are all transformed they are returned by the corresponding > JAX-RS method to the JAX-RS infrastructure. This is encouraged by the > high level interface of JAX-RS, as it encourages you to write methods > like this: > > @GET > Vms get() > > Who could ever have thought that this can be a good idea. :D > I would go with paging instead of streams for that purpose (just make > sure to sort the returned values repeatable). >

This is just what JAX-RS encourages.

> > * Once the JAX-RS infrastructure has the complete result it starts to > convert it to the XML (or JSON) document. > > Same for the reverse direction. > > Put 10K virtual machines instead of just 100 and you see the problem. > > Of course one could use JAX-RS in a way that doesn't have this problem, > it is certainly possible, you can get the output stream and serialize > the result from the resource implementation, in a an streaming fashion. > But then the value of JAX-RS is reduced, you can do the same with a > simpler servlet architecture. That is what I plan to do. > > > Exactly my point. I would do monitoring stuff with different simple > stream servlets. >

Yes, servlet technology is enough and better for that, and for serving the RESTAPI content as well.

> > > > > > > > > > > > > > > > > > > > * Add support for multiple versions of the API, using > > the "Version" > > > > header, and generating different Java classes for > > entities and services. > > > > For example, if we have versions 4 and 5 of the > model as > > separate > > > > artifacts, then we can generate "V4Vm" and "V5Vm" > entity > > classes, and > > > > "V4VmService" and "V5VmService" service classes. These > > can be used > > > > simultaneously in the server, so we can have in > the same > > engine > > > > implementations for multiple versions. > > > > > > > > > > > > There are also many ways to do that. Here [7] is a pretty > > clean way to > > > > do it with jax-rs and you will have everything related in > > one resource. > > > > > > > > > > Yes, there are many ways. In my opinion it is better to use > > the HTTP > > > "Version" header, and to forward requests to different > resource > > > implementations without requiring different URLs or > different > > > content types. > > > > > > Have no strong opinion there, just seemed to be a good choice > > regarding > > > to versioning limitations in jax-rs and our use of jax-rs > > subresources. > > > > > > > > > > > > > > > > > > The final picture isn't completely defined yet. > > > > > > > > Regards, > > > > Juan Hernandez > > > > > > > > > On Mon, Oct 26, 2015 at 4:03 PM, Juan Hernández > > <jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>> > > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>>> > > > > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>> > > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>>>>> wrote: > > > > > > > > > > Hello, > > > > > > > > > > I will soon merge the following patches that > > introduce a new > > > > way to > > > > > specify the contracts of the RESTAPI: > > > > > > > > > > restapi: Introduce metamodel > > > > > https://gerrit.ovirt.org/45852 > > > > > > > > > > restapi: Use metamodel > > > > > https://gerrit.ovirt.org/46478 > > > > > > > > > > restapi: Generate JAX-RS interfaces from model > > > > > https://gerrit.ovirt.org/47337 > > > > > > > > > > > > > > > > > > > Looks pretty much like we are replacing one way of > > > annotating things > > > > > with another way of specifying things. > > > > > Could you elaborate what the benefit of that way of > > > description is? > > > > > > > > > > How would I customize endpoints with e.g. @Gzip > > annotations? > > > Would > > > > I at > > > > > the end still have my JAX-RS annotates resource > classes? > > > > > > > > > > > > > > > These patches introduce a new "metamodel" > concept, > > and move > > > > the current > > > > > specification of the RESTAPI based on XML schema > > and JAX-RS > > > > interfaces > > > > > to a new "model" built on the new metamodel. > > > > > > > > > > > > > > > What does this mean for you in practical terms? > > > Currently when > > > > you want > > > > > to introduce or modify one of the data types > used > > by the > > > > RESTAPI you > > > > > start by modifying the XML schema. Once the > > patches are > > > merged > > > > the XML > > > > > schema will never be touched, as it will be > > automatically > > > > generated from > > > > > the "model". For example, imagine that you > need to > > add a new > > > > "color" > > > > > attribute to the "VM" entity. To do so with > the new > > > model you > > > > will have > > > > > to modify the following file, which is the > > specification of > > > > the "Vm" > > > > > entity, written as a Java interface: > > > > > > > > > > > > > > > > > > > > > > > > > https://gerrit.ovirt.org/#/c/46478/16/backend/manager/modules/restapi/model/... > > > > > > > > > > In that interface you will have to add a > line like > > this: > > > > > > > > > > String color(); > > > > > > > > > > Note that this Java interface is just the > > specification > > > of the > > > > entity, > > > > > it won't be used at all during runtime. > Instead of > > that the > > > > XML schema > > > > > will be generated from it, and then Java will be > > generated > > > > from the XML > > > > > schema, as we do today (this will change in the > > future, but > > > > not yet). > > > > > > > > > > Same for the services. If you want to add a new > > "paint" > > > action > > > > to the > > > > > "Vm" resource then you won't modify the JAX-RS > > interfaces, > > > > instead of > > > > > that you will modify the following file, > which is the > > > > specification of > > > > > the "Vm" service, written as a Java interface: > > > > > > > > > > > > > > > > > > > > > > > > > https://gerrit.ovirt.org/#/c/47337/6/backend/manager/modules/restapi/model/s... > > > > > > > > > > In that interface you will need to add a > sub-interface > > > > representing the > > > > > action: > > > > > > > > > > interface Paint { > > > > > } > > > > > > > > > > The JAX-RS interface will be generated from > that. > > > Currently these > > > > > sub-interfaces are empty. In the future they > will > > > contain the > > > > > specifications of the parameters (currently > in the > > > > rsdl_metadata.yml > > > > > file). > > > > > > > > > > > > > > > > > > > > These changes will currently affect only the > > > specification of the > > > > > RESTAPI, not the implementation, so in in the > > > > "Backend*Resource" classes > > > > > things won't change yet. > > > > > > > > > > > > > > > Currently I do not really understand where we > are going > > > here. Are we > > > > > trying to get rid of rdsl? > > > > > > > > > > So basically two questions: > > > > > > > > > > 1) What is the final goal? > > > > > 2) What speaks agains using Hibernate validator > on Daos in > > > combination > > > > > with JAX-RS annotated resources (and just > removing all > > > interfaces, as > > > > > far as I can see we only have one implementation per > > > endpoint) and > > > > > creating all schemas and clients through SWAGGER > tooling? > > > > > > > > > > > > > > > If you have doubts, please let me know. > > > > > > > > > > Regards, > > > > > Juan Hernandez > > > > > > > > > > -- > > > > > Dirección Comercial: C/Jose Bardasano Baos, > 9, Edif. > > > Gorbea 3, > > > > planta > > > > > 3ºD, 28016 Madrid, Spain > > > > > Inscrita en el Reg. Mercantil de Madrid – C.I.F. > > B82657941 - > > > > Red Hat > > > > > S.L. > > > > > _______________________________________________ > > > > > Devel mailing list > > > > > Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>> > > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>>> > > > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>> > > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>>>> > > > > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>> > > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>>> > > > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>> > > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>>>>> > > > > > http://lists.ovirt.org/mailman/listinfo/devel > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Roman > > > > > > > > > > > > -- > > > > Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. > > Gorbea 3, > > > planta > > > > 3ºD, 28016 Madrid, Spain > > > > Inscrita en el Reg. Mercantil de Madrid – C.I.F. > B82657941 - > > > Red Hat > > > > S.L. > > > > > > > > > > > > > > > > I don't know if it is the right thing to do to invent > > something new > > > > here. I personally would prefer to thread a path which > is very > > > common on > > > > the java community. > > > > I would love follow the DRY principle regarding to the > stack > > and the > > > > code and would just use the great community projects > there. > > > > > > > > It would also completely eliminate any custom magic. > The JAX-RS > > > and CDI > > > > magic is pretty standard and easy to understand. > > > > From my perspective, real JAX-RS resoures have the > advantage of > > > > > > > > * being very easy to understand (there is magic, but the > > > connection to > > > > the real endpoint is pretty clear) > > > > * being easy to customize suff, like adding @GZip to an > > annotation > > > > * describing pretty clearly the connection between the > > generated rest > > > > interface and the internal services > > > > > > > > Finally writing hand crafted tests is also much easier. > > > > > > > > What are your thoughts about that? > > > > > > > > Best Regards, > > > > Roman > > > > > > > > > > > > [1] https://github.com/rmohr/jetty-maven-cdi-demo > > > > [2] > > > > > > > > > > https://github.com/rmohr/jetty-maven-cdi-demo/blob/master/src/main/java/rmoh... > > > > [3] http://swagger.io/ > > > > [4] https://github.com/kongchen/swagger-maven-plugin > > > > [5] https://github.com/swagger-api/swagger-codegen > > > > [6] > > > > > > > > > > https://github.com/rmohr/jetty-maven-cdi-demo/blob/master/src/main/java/rmoh... > > > > [7] > > > > > > > > > > http://maxenglander.com/2013/04/23/basic-restful-api-versioning-in-jersey.ht... > > > > [8] https://github.com/swagger-api/swagger-ui > > > > > > > > > > [9] > > > > > > https://github.com/swagger-api/swagger-core/blob/master/modules/swagger-jaxr... > > > > > >

-- Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta 3ºD, 28016 Madrid, Spain Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.

On Tue, Oct 27, 2015 at 2:41 PM, Juan Hernández <jhernand@redhat.com> wrote:

On 10/27/2015 02:10 PM, Roman Mohr wrote:

...

On Tue, Oct 27, 2015 at 1:45 PM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com>> wrote:

On 10/27/2015 12:55 PM, Roman Mohr wrote: > > > On Tue, Oct 27, 2015 at 12:16 PM, Juan Hernández <

jhernand@redhat.com <mailto:jhernand@redhat.com>

...

> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> wrote: > > On 10/27/2015 11:28 AM, Roman Mohr wrote: > > > > > > On Tue, Oct 27, 2015 at 10:47 AM, Juan Hernández <

jhernand@redhat.com <mailto:jhernand@redhat.com>

...

<mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>> wrote: > > > > On 10/27/2015 10:16 AM, Roman Mohr wrote: > > > > > > > > > On Mon, Oct 26, 2015 at 5:32 PM, Juan Hernández <

jhernand@redhat.com <mailto:jhernand@redhat.com>

...

<mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> > > > <mailto:jhernand@redhat.com <mailto:

jhernand@redhat.com>

...

<mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>>> wrote: > > > > > > On 10/26/2015 04:56 PM, Roman Mohr wrote: > > > > Hi Juan, > > > > > > > > The way to specify the contract look pretty clean and > nice. > > > > I would love to read a few words about the big > picture. What > > is the > > > > final scenario? > > > > > > > > > > The motivation for this change is that currently we > don't have > > a central > > > place where the RESTAPI is specified, rather we have several > > different > > > places, using several different technologies: > > > > > > * XML schema for the data model. > > > * JAX-RS for part of the operational model (without the > > parameters). > > > * rsdl_metadata.yaml for the parameters of the > operational model. > > > > > > This makes it difficult to infer information about

the

...

> model. For > > > example, the generators of the SDKs have to download the XML > > schema, and > > > the RSDL (which is generated from the JAX-RS interfaces > using > > reflection > > > and combining it with the information from the > > rsdl_metadata.yaml file) > > > and then they have to do their own computations to extract > > what they > > > need. > > > > > > Same happens with the CLI: it has to extract the information > > it needs > > > from the Python code generated for the Python SDK,

yet

...

> another > > level of > > > indirection. > > > > > > > > > You are right, that definitely needs to be cleaned up.

I

...

> just want to > > > discuss a few points below with you. > > > > > > > > > > > > We are also lacking a comprehensive reference > documentation of the > > > RESTAPI. What we currently have has been written by > hand, and > > gets out > > > of sync very quickly, and we don't even notice. > > > > > > > > > Did you also consider swagger? It is made for exactly

that

...

> purpose. > > > I created a demo in [1] which uses resteasy, weld, > hibernate-validator > > > and swagger to demonstrate how to do DRY with jaxrs. > > > Would be great to hear you thoughts on that. > > > > > > And there is the great swagger-ui [8] to display the > documentation > > in a > > > more human readable way. > > > > > > > Yes, I considered Swagger, and rejected it because it is JSON > centric, > > and I think JSON isn't as good as Java to represent the > contracts of our > > RESTAPI. > > > > > > You just write plain jax-rs, swagger just creates a description out of > > it. So the source defining the contract is pure java (jax-rs with > some > > swagger annotations for description, etc.). > > Or am I missing the point here? > > > > If I understand correctly the Swagger core is a JSON (or YAML) > specification of the API. From that you can generate JAX-RS annotated > code, not the other way around. So the specification document that you > write is a JSON document. > > > You are right, my terminology here was not clear. Swagger is just a > specification. Swagger-core and swagger-jaxrs are the ones which

can

...

> create the documnetation out of JAX-RS resources. > > > Alternatively, you can use the Swagger annotations to decorate your > implementation, both the entity classes and the JAX-RS resource > implementations, and then extract the model from that. But

this is

...

> putting the implementation before the specification. That is where we > are today, and it causes multiple problems. I think it is better to have > the specification and the implementation separate. Swagger does that > well when using JSON directly, our metamodel also does it well, but > using a better language. > > > > Isn't our problem that we have everything scattered arount the place and > not that we are using JAX-RS? I don't think that this has anything to do > with specification before implementation or implementation before > specification. >

Correct, the problem with the specification if that we don't have

one,

...

just pieces of it scattered around, some in the XML schema, some in

the

...

JAX-RS interfaces, some in the rsdl_metadata.yaml file.

This is historically related with the order of specification and implementation. When the project started we didn't create a specification of the RESTAPI, only an implementation, using JAX-RS

and

...

XML schema. Then we added the rsdl_metadata.yaml file, and extracted

a

...

specification from that, the RSDL. The result is hard to consume. To make it easy to consume we need to reverse the order: first specification, free of implementation details, then implementation.

> > > > > > > In addition we need to do these changes in a smooth way,

without causing

...

> > big changes in the middle. For example, in the first

step we need to

...

> > preserve the JAX-RS interfaces as they are today, to

avoid massive

...

> > changes to all the resource implementations. This could

be done with

...

> > > > Swagger, but would require custom code generators. With

less effort we

...

> > can do our own. > > > > > > This is of course generally a difficult task. But I do not

know why it

...

> > would be more difficult to write a custom swagger reader (if

we even

...

> > have to, it can read the interfaces as well) . > > They are pretty streight forward. Just look at [9], this

contains the

...

> > wole jax-rs specific code to generate the swagger

documentation.

...

> > > > But yes, I don't know every detail here of the engine and

can't clearly

...

> > say that integrating that would just streight forward (my

feeling tells

...

> > me that it would not be too hard). I am just under the

impression that

...

> > we would benefit from that. Just reduces custom magic to a

minimum.

...

> > > > Using something like Swagger would be certainly possible, and

not that

...

> hard, but it requires an effort. For example, say that we

decide to use

...

> the Swagger annotations. Then we will need to add these

annotations to

...

> all our JAX-RS resource implementations. That is a non trivial

effort.

...

> We would need to add the annotations to the entities as well.

But wait,

...

> we don't have such entities, only XML schema. So we would need

a reader

...

> that parses XML schema, and creating it requires effort. > > > You can just create the entities/daos once like we do now on every

build

...

> and annotate them once and drop the whole xml. >

The DAOs aren't part of the RESTAPI. The backend entities aren't

either.

...

The RESTAPI has its own entities which are currently generated from

XML

...

schema.

s/DAOs/DTOs I meant Data transfer object, not Data Access objects. And they are generated.

> > Where do we put > the documentation then? Part in the JAX-RS interfaces, part in

the XML

...

> schema. > > > I don't understand that. There is only one documentation and > implementation source, that is the JAX-RS resource and the

entity/dao

...

> accepted by the endpoint. > The XML schema can just be removed. The documentation in swagger

format

...

> for other tools like swagger-codegen and swagger-ci can be made > available through swagger-maven-plugin or a servlet in the engine

which

...

> generates the swagger json on the fly (like I did in [1]) >

Those are two: JAX-RS and entities, and they are implementations, not specifications.

The XML schema can't be drop, as many clients use it. But we can generate it from the model, and that is what we will do.

Did not look into that.

> > We are already there, and we ended up with no documentation at > all. In my view the sum of these efforts is higher than doing our own > metamodel. > > > > > > > > > > > Swagger UI is certainly great. I did test it and it is really > good. We > > may be able to copy some concepts. > > > > > > > > > > > To solve these issues I intend to have the specification of > > the RESTAPI > > > only in one place, and using only one technology. I > decided to > > use Java > > > interfaces for that. Note however that they are just the > > support for the > > > information, like paper is the support for ink. I decided to > > use Java > > > because it is easy to create, modify and re-factor using > tools > > familiar > > > to most of us. > > > > > > These source of these interfaces is analysed (using QDox, > > currently) and > > > a "model" of the RESTAPI is generated in memory.

This

...

> model is > > > independent of the supporting Java source, and

easy to

...

> > consume. For > > > example, imagine that you want to list all the

types

...

> available > > in the > > > model and for each one display its documentation: > > > > > > Model model = ...; > > > for (Type type : model.getTypes()) { > > > Name name = type.getName(); > > > String doc = type.getDoc(); > > > System.out.println(name + ": " + doc); > > > } > > > > > > Something like this, but more elaborate, will be part of > a web > > > application that provides comprehensive reference > documentation, > > > assuming that we dedicate the time to write documentation > > comments in > > > the specification. > > > > > > I intend to use this model also to do simplify the > generators > > of the > > > SDKs and the CLI. > > > > > > In addition these are some of the things that I

would

...

> like to > > change in > > > the near future (for 4.0): > > > > > > * Move the specification of the parameters of operations out > > of the > > > rsdl_metadata.yaml file and into the model. For example: > > > > > > @Service > > > public VmService { > > > /** > > > * The operation to add a virtual machine. > > > */ > > > interface Add { > > > /** > > > * The representation of the virtual machine

is

...

> received > > > * as parameter, and the representation of the created > > > * virtual machine is returned as result. > > > */ > > > @In @Out Vm vm(); > > > > > > /** > > > * In the future, we will be able to specify other > > > * parameters here. > > > */ > > > @In Boolean force(); > > > > > > /** > > > * Even with default values. > > > */ > > > @In default Boolean force() { return true; } > > > > > > /** > > > * And we will be able to specify constraints, which > > > * will replace the rsdl_metadata.yaml file. > > > */ > > > @Constraint > > > default boolean vmNameMustNotBeNull() { > > > return vm().name() != null; > > > } > > > } > > > } > > > > > > * Enforce the constraints automatically. If the constraints > > are in the > > > model, then we can just check them and reject

requests

...

> before > > delivering > > > them to the application. Currently we do this

manually

...

> (and often > > > forget) with calls to "validate(...)" methods. > > > > > > > > > > > > Did you consider just annotating the DTOs with JSR-303 > annotations and > > > integrate a validator with jax-rs? > > > See [2] for an example. > > > > > > > This is a great way to implement a system, but the goal

here

...

> isn't to > > implement it, rather to specify it. Using annotations in this > way won't > > help the generators of the SDKs, for example, to figure out what > > parameters are required, mandatory, etc. > > > > > > Swagger understands them. From my example project, swagger created > that > > > > description: > > type: "string" > > minLength: 10 > > maxLength: 100 > > > > out of > > > > @Size(min=10, max=100) # jsr-303 > > private String description; > > > > and so does swagger-codegen which can generate clients in

java,

...

> python, ... > > > > This is extracting the specification from the implementation, which > isn't correct in my opinion, it should be the opposite. Not saying that > this makes Swagger bad, it is nice that it has this capability, but I > think we can do it better. > > > In my opition this is the main advantage of that. It is DRY while still > having full control of the implementation. >

I think that DRY is better served if you write the specification

once,

...

and then, from that, you generate the contracts (interfaces,

entities,

...

builders, etc) that the implementation should use.

And how do you add special things like @Gzip or other specializations which you might use?

There shouldn't be any specialization. All the RESTAPI services should behave exactly the same. So if one of them supports GZIP, for example, then all of the should do.

Anyhow, if there is a real need for a specialization it can go into the implementation, it isn't a problem.

...

> > > > > > > > > > * Generate the Java classes directly from the model. Instead of Model -> > > > XML Schema -> Java, we can do Model -> Java. This will allow us to solve > > > some of the XJC compiler limitations, like the horrible way we handle > > > arrays today. > > > > > > > > > Swagger [3] is a rest documentation specification. There is also a maven > > > plugin [4] and you can create clients for example with [5]. > > > > > > > > > > > > * Replace JAX-RS with a simpler infrastructure that supports better > > > streaming and CDI injection. > > > > > > > > > > > > With resteasy-cdi you have pretty good injection support for resteasy. > > > Run the demo in [1] to see it in action and look at the file at [6]. > > > > > > > Resteasy-CDI isn't standard, it only works with Resteasy. If we rely on > > it then we re tied to Resteasy for ever. > > > > > > Even jersey has support for that (I think it is called jeryse-gf-cdi), > > but why would we want switch? I don't think that jboss will

drop

...

> > resteasy and it also works fine outside of full blown containers. I > > don't think that this is an argument. > > > > Well, nobody thought that JBoss would drop Tomcat, and they did. Nobody > thought that Resteasy would change the SPI from 2.x to 3.x, and they > did.\ > > > It will be there for the next year or another library which offers

the

...

> same thing. Having injections in Jax-rs resources is important for > spring, jboss, glassfish and others there will always be ways to do > that. I don't know why we should create our own 'custom standard'

just

...

> because another standard does not include dependency injection

when we

...

> can just extend it so easily. >

Here our views of "easily" are different. I have suffered the

problems

...

with changes in Resteasy over the past years, and I won't describe

it as

...

easy.

I agree. Maven and dependencies are never easy. But I prefer to fix something now and then on regular maven dependency updates than being tied to somesing because of a custom code generator.

> > We want (well, I want) to get out of JAX-RS because it doesn't

support

...

> well CDI and streaming. > > > Why should streaming resources be in the rest interface? >

Streaming should be used not only in the RESTAPI, but in the complete system, including the backend and the DAL, otherwise we have serious problems when clients requests large amounts of objects.

Streaming is no solution here. It does not address the problem at all. Solving the database bottleneck with this approach is just wrong. A client can iterate over our 10000 VMs extremely fast. No matter if streamed or not, so this will still hit the database.

Streaming is a mechanism to improve the latency when serving multiple objects: instead of having your client waiting till you have all the requested objects (10 or 10K) in memory you send them as soon as you can. In addition it also reduces the use of memory in the server.

Think about watching a movie. What is better, downloading it complete and the watch it, or watch as the individual frames arrive?

Don't forget that movies are stored in form which is meant to be streamed. This is not the case when fetching data from databases. Also message brokers use leveldb or other file systems if they have to store something for persistence reasons, which are basically created for streaming.

What you normally do to prevent this:

...

1) A second level cache (see here for a start

https://gerrit.ovirt.org/#/q/status:open+project:ovirt-engine+branch:master+... )

Second level cache doesn't solve the latency problem. If someone requests 1000 VMs he will still have to wait till you have the 1000 VMs arranged in memory before sending them.

...

2) Add paging to the rest interface and define a maximum number per page (would you trottle the stream in your approach per session, request?)

The client then just happily fetch one page after the other without thinking about overloading your system (but of course the client can still deliberately harm you), because a client can assume that you optimize your application (caches, daos and database and page size per request) for that.

The RESTAPI already has a pagination mechanism, but some clients don't want to use it. We used to have a hard limit and we had to remove it.

...

Of course for notifications streams are great, but again why should they be in the rest interface? Adding websockets or other stuff to listen on changed VMs would be great. Just discussed this recently with rgolan.

Notifications are currently out of the scope of the RESTAPI, and there are no plans to add them.

...

Think, for example, that a client requests 100 virtual machines.

What we

...

currently do is the following:

* The DAL requests the 100 rows to the database, and waits till the

100

...

rows are loaded in memory (this is how the JDBC driver works by default). Only when the 100 rows are in memory the are converted to backend objects.

* The DAL returns the 100 backend objects to the BLL, which processes them. Only when they are all processed they are returned to the

RESTAPI,

...

as a list.

* The RESTAPI transforms the 100 backend objects into RESTAPI

objects.

...

Only when the are all transformed they are returned by the

corresponding

...

JAX-RS method to the JAX-RS infrastructure. This is encouraged by the high level interface of JAX-RS, as it encourages you to write methods like this:

@GET Vms get()

Who could ever have thought that this can be a good idea. :D I would go with paging instead of streams for that purpose (just make sure to sort the returned values repeatable).

This is just what JAX-RS encourages.

...

* Once the JAX-RS infrastructure has the complete result it starts to convert it to the XML (or JSON) document.

Same for the reverse direction.

Put 10K virtual machines instead of just 100 and you see the problem.

Of course one could use JAX-RS in a way that doesn't have this

problem,

...

it is certainly possible, you can get the output stream and serialize the result from the resource implementation, in a an streaming

fashion.

...

But then the value of JAX-RS is reduced, you can do the same with a simpler servlet architecture. That is what I plan to do.

Exactly my point. I would do monitoring stuff with different simple stream servlets.

Yes, servlet technology is enough and better for that, and for serving the RESTAPI content as well.

...

> > > > > > > > > > > > * Add support for multiple versions of the API,

using

...

> the "Version" > > > header, and generating different Java classes for > entities and services. > > > For example, if we have versions 4 and 5 of the model as > separate > > > artifacts, then we can generate "V4Vm" and "V5Vm" entity > classes, and > > > "V4VmService" and "V5VmService" service classes.

These

...

> can be used > > > simultaneously in the server, so we can have in the same > engine > > > implementations for multiple versions. > > > > > > > > > There are also many ways to do that. Here [7] is a

pretty

...

> clean way to > > > do it with jax-rs and you will have everything related

in

...

> one resource. > > > > > > > Yes, there are many ways. In my opinion it is better to

use

...

> the HTTP > > "Version" header, and to forward requests to different resource > > implementations without requiring different URLs or different > > content types. > > > > Have no strong opinion there, just seemed to be a good choice > regarding > > to versioning limitations in jax-rs and our use of jax-rs > subresources. > > > > > > > > > > > > > The final picture isn't completely defined yet. > > > > > > Regards, > > > Juan Hernandez > > > > > > > On Mon, Oct 26, 2015 at 4:03 PM, Juan Hernández > <jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>> > > > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>>>> wrote: > > > > > > > > Hello, > > > > > > > > I will soon merge the following patches that > introduce a new > > > way to > > > > specify the contracts of the RESTAPI: > > > > > > > > restapi: Introduce metamodel > > > > https://gerrit.ovirt.org/45852 > > > > > > > > restapi: Use metamodel > > > > https://gerrit.ovirt.org/46478 > > > > > > > > restapi: Generate JAX-RS interfaces from

model

...

> > > > https://gerrit.ovirt.org/47337 > > > > > > > > > > > > > > > Looks pretty much like we are replacing one way

of

...

> > annotating things > > > > with another way of specifying things. > > > > Could you elaborate what the benefit of that way

of

...

> > description is? > > > > > > > > How would I customize endpoints with e.g. @Gzip > annotations? > > Would > > > I at > > > > the end still have my JAX-RS annotates resource classes? > > > > > > > > > > > > These patches introduce a new "metamodel" concept, > and move > > > the current > > > > specification of the RESTAPI based on XML

schema

...

> and JAX-RS > > > interfaces > > > > to a new "model" built on the new metamodel. > > > > > > > > > > > > What does this mean for you in practical

terms?

...

> > Currently when > > > you want > > > > to introduce or modify one of the data types used > by the > > > RESTAPI you > > > > start by modifying the XML schema. Once the > patches are > > merged > > > the XML > > > > schema will never be touched, as it will be > automatically > > > generated from > > > > the "model". For example, imagine that you need to > add a new > > > "color" > > > > attribute to the "VM" entity. To do so with the new > > model you > > > will have > > > > to modify the following file, which is the > specification of > > > the "Vm" > > > > entity, written as a Java interface: > > > > > > > > > > > > > > > > > >

https://gerrit.ovirt.org/#/c/46478/16/backend/manager/modules/restapi/model/...

...

> > > > > > > > In that interface you will have to add a line like > this: > > > > > > > > String color(); > > > > > > > > Note that this Java interface is just the > specification > > of the > > > entity, > > > > it won't be used at all during runtime. Instead of > that the > > > XML schema > > > > will be generated from it, and then Java

will be

...

> generated > > > from the XML > > > > schema, as we do today (this will change in

the

...

> future, but > > > not yet). > > > > > > > > Same for the services. If you want to add a

new

...

> "paint" > > action > > > to the > > > > "Vm" resource then you won't modify the

JAX-RS

...

> interfaces, > > > instead of > > > > that you will modify the following file, which is the > > > specification of > > > > the "Vm" service, written as a Java

interface:

...

> > > > > > > > > > > > > > > > > >

https://gerrit.ovirt.org/#/c/47337/6/backend/manager/modules/restapi/model/s...

...

> > > > > > > > In that interface you will need to add a sub-interface > > > representing the > > > > action: > > > > > > > > interface Paint { > > > > } > > > > > > > > The JAX-RS interface will be generated from that. > > Currently these > > > > sub-interfaces are empty. In the future they will > > contain the > > > > specifications of the parameters (currently in the > > > rsdl_metadata.yml > > > > file). > > > > > > > > > > > > > > > > These changes will currently affect only the > > specification of the > > > > RESTAPI, not the implementation, so in in the > > > "Backend*Resource" classes > > > > things won't change yet. > > > > > > > > > > > > Currently I do not really understand where we are going > > here. Are we > > > > trying to get rid of rdsl? > > > > > > > > So basically two questions: > > > > > > > > 1) What is the final goal? > > > > 2) What speaks agains using Hibernate validator on Daos in > > combination > > > > with JAX-RS annotated resources (and just removing all > > interfaces, as > > > > far as I can see we only have one implementation

per

...

> > endpoint) and > > > > creating all schemas and clients through SWAGGER tooling? > > > > > > > > > > > > If you have doubts, please let me know. > > > > > > > > Regards, > > > > Juan Hernandez > > > > > > > > -- > > > > Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. > > Gorbea 3, > > > planta > > > > 3ºD, 28016 Madrid, Spain > > > > Inscrita en el Reg. Mercantil de Madrid –

C.I.F.

...

> B82657941 - > > > Red Hat > > > > S.L. > > > >

_______________________________________________

...

> > > > Devel mailing list > > > > Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>> > > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>>> > > > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>> > > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>>>> > > > >

http://lists.ovirt.org/mailman/listinfo/devel

...

> > > > > > > > > > > > Thanks, > > > > > > > > Roman > > > > > > > > > -- > > > Dirección Comercial: C/Jose Bardasano Baos, 9,

Edif.

...

> Gorbea 3, > > planta > > > 3ºD, 28016 Madrid, Spain > > > Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - > > Red Hat > > > S.L. > > > > > > > > > > > > I don't know if it is the right thing to do to invent > something new > > > here. I personally would prefer to thread a path which is very > > common on > > > the java community. > > > I would love follow the DRY principle regarding to the stack > and the > > > code and would just use the great community projects there. > > > > > > It would also completely eliminate any custom magic. The JAX-RS > > and CDI > > > magic is pretty standard and easy to understand. > > > From my perspective, real JAX-RS resoures have the advantage of > > > > > > * being very easy to understand (there is magic, but

the

...

> > connection to > > > the real endpoint is pretty clear) > > > * being easy to customize suff, like adding @GZip to

an

...

> annotation > > > * describing pretty clearly the connection between the > generated rest > > > interface and the internal services > > > > > > Finally writing hand crafted tests is also much easier. > > > > > > What are your thoughts about that? > > > > > > Best Regards, > > > Roman > > > > > > > > > [1] https://github.com/rmohr/jetty-maven-cdi-demo > > > [2] > > > > > >

https://github.com/rmohr/jetty-maven-cdi-demo/blob/master/src/main/java/rmoh...

...

> > > [3] http://swagger.io/ > > > [4] https://github.com/kongchen/swagger-maven-plugin > > > [5] https://github.com/swagger-api/swagger-codegen > > > [6] > > > > > >

https://github.com/rmohr/jetty-maven-cdi-demo/blob/master/src/main/java/rmoh...

...

> > > [7] > > > > > >

http://maxenglander.com/2013/04/23/basic-restful-api-versioning-in-jersey.ht...

...

> > > [8] https://github.com/swagger-api/swagger-ui > > > > > > > [9] > > >

https://github.com/swagger-api/swagger-core/blob/master/modules/swagger-jaxr...

...

> > >

-- Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta 3ºD, 28016 Madrid, Spain Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.

On Tue, Oct 27, 2015 at 12:16 PM, Juan Hernández <jhernand@redhat.com> wrote:

On 10/27/2015 11:28 AM, Roman Mohr wrote:

...

On Tue, Oct 27, 2015 at 10:47 AM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com>> wrote:

On 10/27/2015 10:16 AM, Roman Mohr wrote: > > > On Mon, Oct 26, 2015 at 5:32 PM, Juan Hernández <

jhernand@redhat.com <mailto:jhernand@redhat.com>

...

> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> wrote: > > On 10/26/2015 04:56 PM, Roman Mohr wrote: > > Hi Juan, > > > > The way to specify the contract look pretty clean and nice. > > I would love to read a few words about the big picture. What is the > > final scenario? > > > > The motivation for this change is that currently we don't have a central > place where the RESTAPI is specified, rather we have several different > places, using several different technologies: > > * XML schema for the data model. > * JAX-RS for part of the operational model (without the parameters). > * rsdl_metadata.yaml for the parameters of the operational

model.

...

> > This makes it difficult to infer information about the model.

For

...

> example, the generators of the SDKs have to download the XML schema, and > the RSDL (which is generated from the JAX-RS interfaces using reflection > and combining it with the information from the rsdl_metadata.yaml file) > and then they have to do their own computations to extract what they > need. > > Same happens with the CLI: it has to extract the information it needs > from the Python code generated for the Python SDK, yet another level of > indirection. > > > You are right, that definitely needs to be cleaned up. I just want

to

...

> discuss a few points below with you. > > > > We are also lacking a comprehensive reference documentation of

the

...

> RESTAPI. What we currently have has been written by hand, and gets out > of sync very quickly, and we don't even notice. > > > Did you also consider swagger? It is made for exactly that purpose. > I created a demo in [1] which uses resteasy, weld,

hibernate-validator

...

> and swagger to demonstrate how to do DRY with jaxrs. > Would be great to hear you thoughts on that. > > And there is the great swagger-ui [8] to display the documentation in a > more human readable way. >

Yes, I considered Swagger, and rejected it because it is JSON

centric,

...

and I think JSON isn't as good as Java to represent the contracts of

our

...

RESTAPI.

You just write plain jax-rs, swagger just creates a description out of it. So the source defining the contract is pure java (jax-rs with some swagger annotations for description, etc.). Or am I missing the point here?

If I understand correctly the Swagger core is a JSON (or YAML) specification of the API. From that you can generate JAX-RS annotated code, not the other way around. So the specification document that you write is a JSON document.

Alternatively, you can use the Swagger annotations to decorate your implementation, both the entity classes and the JAX-RS resource implementations, and then extract the model from that. But this is putting the implementation before the specification. That is where we are today, and it causes multiple problems. I think it is better to have the specification and the implementation separate. Swagger does that well when using JSON directly, our metamodel also does it well, but using a better language.

...

In addition we need to do these changes in a smooth way, without

causing

...

big changes in the middle. For example, in the first step we need to preserve the JAX-RS interfaces as they are today, to avoid massive changes to all the resource implementations. This could be done with

Swagger, but would require custom code generators. With less effort

we

...

can do our own.

This is of course generally a difficult task. But I do not know why it would be more difficult to write a custom swagger reader (if we even have to, it can read the interfaces as well) . They are pretty streight forward. Just look at [9], this contains the wole jax-rs specific code to generate the swagger documentation.

But yes, I don't know every detail here of the engine and can't clearly say that integrating that would just streight forward (my feeling tells me that it would not be too hard). I am just under the impression that we would benefit from that. Just reduces custom magic to a minimum.

Using something like Swagger would be certainly possible, and not that hard, but it requires an effort. For example, say that we decide to use the Swagger annotations. Then we will need to add these annotations to all our JAX-RS resource implementations. That is a non trivial effort. We would need to add the annotations to the entities as well. But wait, we don't have such entities, only XML schema. So we would need a reader that parses XML schema, and creating it requires effort. Where do we put the documentation then? Part in the JAX-RS interfaces, part in the XML schema. We are already there, and we ended up with no documentation at all. In my view the sum of these efforts is higher than doing our own metamodel.

...

Swagger UI is certainly great. I did test it and it is really good.

We

...

may be able to copy some concepts.

> > > To solve these issues I intend to have the specification of the RESTAPI > only in one place, and using only one technology. I decided to use Java > interfaces for that. Note however that they are just the support for the > information, like paper is the support for ink. I decided to use Java > because it is easy to create, modify and re-factor using tools familiar > to most of us. > > These source of these interfaces is analysed (using QDox, currently) and > a "model" of the RESTAPI is generated in memory. This model is > independent of the supporting Java source, and easy to consume. For > example, imagine that you want to list all the types available in the > model and for each one display its documentation: > > Model model = ...; > for (Type type : model.getTypes()) { > Name name = type.getName(); > String doc = type.getDoc(); > System.out.println(name + ": " + doc); > } > > Something like this, but more elaborate, will be part of a web > application that provides comprehensive reference

documentation,

...

> assuming that we dedicate the time to write documentation comments in > the specification. > > I intend to use this model also to do simplify the generators of the > SDKs and the CLI. > > In addition these are some of the things that I would like to change in > the near future (for 4.0): > > * Move the specification of the parameters of operations out of the > rsdl_metadata.yaml file and into the model. For example: > > @Service > public VmService { > /** > * The operation to add a virtual machine. > */ > interface Add { > /** > * The representation of the virtual machine is received > * as parameter, and the representation of the created > * virtual machine is returned as result. > */ > @In @Out Vm vm(); > > /** > * In the future, we will be able to specify other > * parameters here. > */ > @In Boolean force(); > > /** > * Even with default values. > */ > @In default Boolean force() { return true; } > > /** > * And we will be able to specify constraints, which > * will replace the rsdl_metadata.yaml file. > */ > @Constraint > default boolean vmNameMustNotBeNull() { > return vm().name() != null; > } > } > } > > * Enforce the constraints automatically. If the constraints are in the > model, then we can just check them and reject requests before delivering > them to the application. Currently we do this manually (and

often

...

> forget) with calls to "validate(...)" methods. > > > > Did you consider just annotating the DTOs with JSR-303 annotations

and

...

> integrate a validator with jax-rs? > See [2] for an example. >

This is a great way to implement a system, but the goal here isn't to implement it, rather to specify it. Using annotations in this way

won't

...

help the generators of the SDKs, for example, to figure out what parameters are required, mandatory, etc.

Swagger understands them. From my example project, swagger created that

description: type: "string" minLength: 10 maxLength: 100

out of

@Size(min=10, max=100) # jsr-303 private String description;

and so does swagger-codegen which can generate clients in java, python,

...

...

This is extracting the specification from the implementation, which isn't correct in my opinion, it should be the opposite. Not saying that this makes Swagger bad, it is nice that it has this capability, but I think we can do it better.

...

> > > * Generate the Java classes directly from the model. Instead

of Model ->

...

> XML Schema -> Java, we can do Model -> Java. This will allow

us to solve

...

> some of the XJC compiler limitations, like the horrible way we

handle

...

> arrays today. > > > Swagger [3] is a rest documentation specification. There is also a

maven

...

> plugin [4] and you can create clients for example with [5]. > > > > * Replace JAX-RS with a simpler infrastructure that supports

better

...

> streaming and CDI injection. > > > > With resteasy-cdi you have pretty good injection support for

resteasy.

...

> Run the demo in [1] to see it in action and look at the file at

[6].

...

>

Resteasy-CDI isn't standard, it only works with Resteasy. If we rely

on

...

it then we re tied to Resteasy for ever.

Even jersey has support for that (I think it is called jeryse-gf-cdi), but why would we want switch? I don't think that jboss will drop resteasy and it also works fine outside of full blown containers. I don't think that this is an argument.

Well, nobody thought that JBoss would drop Tomcat, and they did. Nobody thought that Resteasy would change the SPI from 2.x to 3.x, and they did.

We want (well, I want) to get out of JAX-RS because it doesn't support well CDI and streaming.

By the way why do you think it does not support streaming well? As far as I know there are StreamingOutputs for specific endpoints and you can always plug in your own MessageBodyWriter which can serialize item by item of a list to return everything in a stream like fashion. If you use lazy evaluating collections like guava does for a long time for many things we would be pretty much do what you are after, right? JAX-RS is not the problem here, just standard jackson message body writers do not do it.

...

> > > * Add support for multiple versions of the API, using the

"Version"

...

> header, and generating different Java classes for entities and

services.

...

> For example, if we have versions 4 and 5 of the model as

separate

...

> artifacts, then we can generate "V4Vm" and "V5Vm" entity

classes, and

...

> "V4VmService" and "V5VmService" service classes. These can be

used

...

> simultaneously in the server, so we can have in the same engine > implementations for multiple versions. > > > There are also many ways to do that. Here [7] is a pretty clean

way to

...

> do it with jax-rs and you will have everything related in one

resource.

...

>

Yes, there are many ways. In my opinion it is better to use the HTTP "Version" header, and to forward requests to different resource implementations without requiring different URLs or different content types.

Have no strong opinion there, just seemed to be a good choice regarding to versioning limitations in jax-rs and our use of jax-rs subresources.

> > > The final picture isn't completely defined yet. > > Regards, > Juan Hernandez > > > On Mon, Oct 26, 2015 at 4:03 PM, Juan Hernández <

jhernand@redhat.com <mailto:jhernand@redhat.com>

...

<mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>> > > <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>>> wrote: > > > > Hello, > > > > I will soon merge the following patches that introduce a

new

...

> way to > > specify the contracts of the RESTAPI: > > > > restapi: Introduce metamodel > > https://gerrit.ovirt.org/45852 > > > > restapi: Use metamodel > > https://gerrit.ovirt.org/46478 > > > > restapi: Generate JAX-RS interfaces from model > > https://gerrit.ovirt.org/47337 > > > > > > > Looks pretty much like we are replacing one way of annotating things > > with another way of specifying things. > > Could you elaborate what the benefit of that way of description is? > > > > How would I customize endpoints with e.g. @Gzip annotations? Would > I at > > the end still have my JAX-RS annotates resource classes? > > > > > > These patches introduce a new "metamodel" concept, and

move

...

> the current > > specification of the RESTAPI based on XML schema and

JAX-RS

...

> interfaces > > to a new "model" built on the new metamodel. > > > > > > What does this mean for you in practical terms? Currently when > you want > > to introduce or modify one of the data types used by the > RESTAPI you > > start by modifying the XML schema. Once the patches are merged > the XML > > schema will never be touched, as it will be automatically > generated from > > the "model". For example, imagine that you need to add a

new

...

> "color" > > attribute to the "VM" entity. To do so with the new model you > will have > > to modify the following file, which is the specification

of

...

> the "Vm" > > entity, written as a Java interface: > > > > > > >

https://gerrit.ovirt.org/#/c/46478/16/backend/manager/modules/restapi/model/...

...

> > > > In that interface you will have to add a line like this: > > > > String color(); > > > > Note that this Java interface is just the specification of the > entity, > > it won't be used at all during runtime. Instead of that

the

...

> XML schema > > will be generated from it, and then Java will be

generated

...

> from the XML > > schema, as we do today (this will change in the future,

but

...

> not yet). > > > > Same for the services. If you want to add a new "paint" action > to the > > "Vm" resource then you won't modify the JAX-RS

interfaces,

...

> instead of > > that you will modify the following file, which is the > specification of > > the "Vm" service, written as a Java interface: > > > > > > >

https://gerrit.ovirt.org/#/c/47337/6/backend/manager/modules/restapi/model/s...

...

> > > > In that interface you will need to add a sub-interface > representing the > > action: > > > > interface Paint { > > } > > > > The JAX-RS interface will be generated from that. Currently these > > sub-interfaces are empty. In the future they will contain the > > specifications of the parameters (currently in the > rsdl_metadata.yml > > file). > > > > > > > > These changes will currently affect only the specification of the > > RESTAPI, not the implementation, so in in the > "Backend*Resource" classes > > things won't change yet. > > > > > > Currently I do not really understand where we are going here. Are we > > trying to get rid of rdsl? > > > > So basically two questions: > > > > 1) What is the final goal? > > 2) What speaks agains using Hibernate validator on Daos in combination > > with JAX-RS annotated resources (and just removing all interfaces, as > > far as I can see we only have one implementation per endpoint) and > > creating all schemas and clients through SWAGGER tooling? > > > > > > If you have doubts, please let me know. > > > > Regards, > > Juan Hernandez > > > > -- > > Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, > planta > > 3ºD, 28016 Madrid, Spain > > Inscrita en el Reg. Mercantil de Madrid – C.I.F.

B82657941 -

...

> Red Hat > > S.L. > > _______________________________________________ > > Devel mailing list > > Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>> > <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org> <mailto:Devel@ovirt.org <mailto:Devel@ovirt.org>>> > > http://lists.ovirt.org/mailman/listinfo/devel > > > > > > Thanks, > > > > Roman > > > -- > Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta > 3ºD, 28016 Madrid, Spain > Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat > S.L. > > > > I don't know if it is the right thing to do to invent something new > here. I personally would prefer to thread a path which is very common on > the java community. > I would love follow the DRY principle regarding to the stack and

the

...

> code and would just use the great community projects there. > > It would also completely eliminate any custom magic. The JAX-RS and CDI > magic is pretty standard and easy to understand. > From my perspective, real JAX-RS resoures have the advantage of > > * being very easy to understand (there is magic, but the connection to > the real endpoint is pretty clear) > * being easy to customize suff, like adding @GZip to an annotation > * describing pretty clearly the connection between the generated

rest

...

> interface and the internal services > > Finally writing hand crafted tests is also much easier. > > What are your thoughts about that? > > Best Regards, > Roman > > > [1] https://github.com/rmohr/jetty-maven-cdi-demo > [2] >

https://github.com/rmohr/jetty-maven-cdi-demo/blob/master/src/main/java/rmoh...

...

> [3] http://swagger.io/ > [4] https://github.com/kongchen/swagger-maven-plugin > [5] https://github.com/swagger-api/swagger-codegen > [6] >

https://github.com/rmohr/jetty-maven-cdi-demo/blob/master/src/main/java/rmoh...

...

> [7] >

http://maxenglander.com/2013/04/23/basic-restful-api-versioning-in-jersey.ht...

...

> [8] https://github.com/swagger-api/swagger-ui >

[9]

https://github.com/swagger-api/swagger-core/blob/master/modules/swagger-jaxr...

...

-- Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta 3ºD, 28016 Madrid, Spain Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.