Here's a draft wiki of the libosinfo integration with the engine http://wiki.ovirt.org/wiki/Libosinfo please share your comments on this thread and I'll update the wiki accordingly.

From: "Juan Hernandez" <jhernand(a)redhat.com&gt; To: "Daniel P. Berrange" <berrange(a)redhat.com&gt; Cc: "arch" <arch(a)ovirt.org&gt;, engine-devel(a)ovirt.org Sent: Monday, September 24, 2012 6:30:43 PM Subject: Re: [Engine-devel] libosinfo integration On 09/24/2012 05:24 PM, Daniel P. Berrange wrote: > On Mon, Sep 24, 2012 at 04:51:37PM +0200, Roy Golan wrote: >> Here's a draft wiki of the libosinfo integration with the engine >> http://wiki.ovirt.org/wiki/Libosinfo >> >> please share your comments on this thread and I'll update the wiki >> accordingly. > > As a libosinfo maintainer I strongly discourage you from writing a > custom > API directly against the XML. While the format is stable from the > POV > that vendors/users can create new XML docs for new/customized OS > variants > they have, we do not consider it something for applications to be > using > directly. The libosinfo library API includes alot of logic beyond > simply > being an convenient way to process the XML. In particular dealing > with > the inheritance of data across OS, searching for data across a > variety > of data sources, relying on GIO for URI resolving, live update of > the > database, and more. > > We're also not really interested in maintaining manually written > bindings > for any languages. Rather than putting effort into such bindings > for Java, > time is better spent on finishing the GOBject introspection mapping > layer > for Java. There was a proof of concept, which was never finished / > updated > to the final introspection spec, which can be used as a basis for > ongoing > development. > > https://live.gnome.org/JGIR > > This work will be applicable way beyond just libosinfo, to a wide > variety of useful APIs. Of possible interest to VDSM in the future > will > be things like the new libvirt-designer & libvirt-builder APIs > which > are going to provide a bridge between libvirt + libosinfo to simply > life for application developers constructing XML for provisioning > guests. I think that the use of JGIR has already been discussed as a way to interface the engine with libvdsm.so: http://lists.ovirt.org/pipermail/arch/2012-August/000760.html If I remember correctly the outcome of that discussion was that it shouldn't be used because it is considered dangerous to use native libraries from Java and because JGIR itself is very outdated. I don't 100% agree with the first part, but the second part is completely true. Anyhow, as it seems to be the right way to interface with libosinfo and future libvirt capabilities, we may want to rethink and put some effort to bring it up to date.

From: "Juan Hernandez" <jhernand(a)redhat.com&gt; To: "Alon Bar-Lev" <alonbl(a)redhat.com&gt; Cc: "arch" <arch(a)ovirt.org&gt;, engine-devel(a)ovirt.org, "Daniel P. Berrange" <berrange(a)redhat.com&gt; Sent: Monday, September 24, 2012 9:33:38 PM Subject: Re: [Engine-devel] libosinfo integration On 09/24/2012 08:59 PM, Alon Bar-Lev wrote: > > > ----- Original Message ----- >> From: "Juan Hernandez" <jhernand(a)redhat.com&gt; >> To: "Daniel P. Berrange" <berrange(a)redhat.com&gt; >> Cc: "arch" <arch(a)ovirt.org&gt;, engine-devel(a)ovirt.org >> Sent: Monday, September 24, 2012 6:30:43 PM >> Subject: Re: [Engine-devel] libosinfo integration >> >> On 09/24/2012 05:24 PM, Daniel P. Berrange wrote: >>> On Mon, Sep 24, 2012 at 04:51:37PM +0200, Roy Golan wrote: >>>> Here's a draft wiki of the libosinfo integration with the engine >>>> http://wiki.ovirt.org/wiki/Libosinfo >>>> >>>> please share your comments on this thread and I'll update the >>>> wiki >>>> accordingly. >>> >>> As a libosinfo maintainer I strongly discourage you from writing >>> a >>> custom >>> API directly against the XML. While the format is stable from the >>> POV >>> that vendors/users can create new XML docs for new/customized OS >>> variants >>> they have, we do not consider it something for applications to be >>> using >>> directly. The libosinfo library API includes alot of logic beyond >>> simply >>> being an convenient way to process the XML. In particular dealing >>> with >>> the inheritance of data across OS, searching for data across a >>> variety >>> of data sources, relying on GIO for URI resolving, live update of >>> the >>> database, and more. >>> >>> We're also not really interested in maintaining manually written >>> bindings >>> for any languages. Rather than putting effort into such bindings >>> for Java, >>> time is better spent on finishing the GOBject introspection >>> mapping >>> layer >>> for Java. There was a proof of concept, which was never finished >>> / >>> updated >>> to the final introspection spec, which can be used as a basis for >>> ongoing >>> development. >>> >>> https://live.gnome.org/JGIR >>> >>> This work will be applicable way beyond just libosinfo, to a wide >>> variety of useful APIs. Of possible interest to VDSM in the >>> future >>> will >>> be things like the new libvirt-designer & libvirt-builder APIs >>> which >>> are going to provide a bridge between libvirt + libosinfo to >>> simply >>> life for application developers constructing XML for provisioning >>> guests. >> >> I think that the use of JGIR has already been discussed as a way >> to >> interface the engine with libvdsm.so: >> >> http://lists.ovirt.org/pipermail/arch/2012-August/000760.html >> >> If I remember correctly the outcome of that discussion was that it >> shouldn't be used because it is considered dangerous to use native >> libraries from Java and because JGIR itself is very outdated. I >> don't >> 100% agree with the first part, but the second part is completely >> true. >> >> Anyhow, as it seems to be the right way to interface with >> libosinfo >> and >> future libvirt capabilities, we may want to rethink and put some >> effort >> to bring it up to date. > > I don't think Java should go native to obtain data, using JNI or > whatever native interface that is offered. It can totally break > the JVM and cause leaks and stability issues JRE would like to > avoid. This is were I disagree, only a bit. It is true that using JNI *without care* can totally break the JVM, generate leaks, and bring down all the application. But if the JNI layer is developed carefully this should not be more risky than any of the other multiple native libraries that you use every day from Java without noticing. In addition JGIR is based on JNA (which is well maintained and stable) and doesn't include any "dangerous" native C code.

> Having language binding for data access logic seems a bit strange > for me, as it should be easy to stabilize a schema either using > XML or a set of CSVs, for all to read and use. > > I do understand why JNI is to be used when OS specific features are > to be used, for example unix domain sockets, shared memory, > security, but not for data access. > > The alternatives would be: > > 1. Run a python program at ovirt-engine start-up, use the python > API to fetch the data we need, and generate our own tiny XML of > the subset or even load it to the database. We can use the output > safely in Java. Disadvantage is that we need to stop/start service > or have a "refresh" verb for reload after libosinfo update. > > 2. When data is requested, execute a python process with cmdline > arguments and get result from stdout of the process. The > disadvantage is the overhead of clone(). > > 3. Read the XML directly on runtime, disadvantage is if XML is > changed. > > 4. Use the libosinfo python API during build and distribute an > update to the information in versions of ovirt, disadvantage is > slow refresh of data, not sure how often this data is updated. > > 5. Provide a web service that has specific XML-RPC / RESTful API, > that can provide data out of the libosinfo, and publish it at > internet. Configure ovirt to use it, disadvantage is the need for > internet connection. > > 6. Configure cgi-bin locally at apache where ovirt is running, > which provide XML-RPC / RESTful API, and access it locally. > > In any case I would avoid exiting the JVM to interact with data > provider. > > Regards, > Alon. > -- Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta 3ºD, 28016 Madrid, Spain Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.

From: "Roy Golan" <rgolan(a)redhat.com&gt; To: engine-devel(a)ovirt.org Sent: Thursday, October 11, 2012 12:22:51 PM Subject: Re: [Engine-devel] libosinfo integration After considering the pros and cons and making poc's I've decided to take the direct XML and java pure implementation. http://gerrit.ovirt.org/#/c/8472/ http://gerrit.ovirt.org/#/c/8473/ the main motivation was is that the XML loading was fast and standard. it is very easy to expose a subset of the API while leaving behind the extra library magic which we don't currently need. Maybe at some point the inheritance feature will come in handy and the jgir implementation will be done so we could switch to it (I'm really not sure whats it there which is not complete)

On 09/24/2012 10:01 PM, Alon Bar-Lev wrote: > > ----- Original Message ----- >> From: "Juan Hernandez" <jhernand(a)redhat.com&gt; >> To: "Alon Bar-Lev" <alonbl(a)redhat.com&gt; >> Cc: "arch" <arch(a)ovirt.org&gt;, engine-devel(a)ovirt.org, "Daniel P. >> Berrange" <berrange(a)redhat.com&gt; >> Sent: Monday, September 24, 2012 9:33:38 PM >> Subject: Re: [Engine-devel] libosinfo integration >> >> On 09/24/2012 08:59 PM, Alon Bar-Lev wrote: >>> >>> ----- Original Message ----- >>>> From: "Juan Hernandez" <jhernand(a)redhat.com&gt; >>>> To: "Daniel P. Berrange" <berrange(a)redhat.com&gt; >>>> Cc: "arch" <arch(a)ovirt.org&gt;, engine-devel(a)ovirt.org >>>> Sent: Monday, September 24, 2012 6:30:43 PM >>>> Subject: Re: [Engine-devel] libosinfo integration >>>> >>>> On 09/24/2012 05:24 PM, Daniel P. Berrange wrote: >>>>> On Mon, Sep 24, 2012 at 04:51:37PM +0200, Roy Golan wrote: >>>>>> Here's a draft wiki of the libosinfo integration with the >>>>>> engine >>>>>> http://wiki.ovirt.org/wiki/Libosinfo >>>>>> >>>>>> please share your comments on this thread and I'll update the >>>>>> wiki >>>>>> accordingly. >>>>> As a libosinfo maintainer I strongly discourage you from >>>>> writing >>>>> a >>>>> custom >>>>> API directly against the XML. While the format is stable from >>>>> the >>>>> POV >>>>> that vendors/users can create new XML docs for new/customized >>>>> OS >>>>> variants >>>>> they have, we do not consider it something for applications to >>>>> be >>>>> using >>>>> directly. The libosinfo library API includes alot of logic >>>>> beyond >>>>> simply >>>>> being an convenient way to process the XML. In particular >>>>> dealing >>>>> with >>>>> the inheritance of data across OS, searching for data across a >>>>> variety >>>>> of data sources, relying on GIO for URI resolving, live update >>>>> of >>>>> the >>>>> database, and more. >>>>> >>>>> We're also not really interested in maintaining manually >>>>> written >>>>> bindings >>>>> for any languages. Rather than putting effort into such >>>>> bindings >>>>> for Java, >>>>> time is better spent on finishing the GOBject introspection >>>>> mapping >>>>> layer >>>>> for Java. There was a proof of concept, which was never >>>>> finished >>>>> / >>>>> updated >>>>> to the final introspection spec, which can be used as a basis >>>>> for >>>>> ongoing >>>>> development. >>>>> >>>>> https://live.gnome.org/JGIR >>>>> >>>>> This work will be applicable way beyond just libosinfo, to a >>>>> wide >>>>> variety of useful APIs. Of possible interest to VDSM in the >>>>> future >>>>> will >>>>> be things like the new libvirt-designer & libvirt-builder APIs >>>>> which >>>>> are going to provide a bridge between libvirt + libosinfo to >>>>> simply >>>>> life for application developers constructing XML for >>>>> provisioning >>>>> guests. >>>> I think that the use of JGIR has already been discussed as a way >>>> to >>>> interface the engine with libvdsm.so: >>>> >>>> http://lists.ovirt.org/pipermail/arch/2012-August/000760.html >>>> >>>> If I remember correctly the outcome of that discussion was that >>>> it >>>> shouldn't be used because it is considered dangerous to use >>>> native >>>> libraries from Java and because JGIR itself is very outdated. I >>>> don't >>>> 100% agree with the first part, but the second part is >>>> completely >>>> true. >>>> >>>> Anyhow, as it seems to be the right way to interface with >>>> libosinfo >>>> and >>>> future libvirt capabilities, we may want to rethink and put some >>>> effort >>>> to bring it up to date. >>> I don't think Java should go native to obtain data, using JNI or >>> whatever native interface that is offered. It can totally break >>> the JVM and cause leaks and stability issues JRE would like to >>> avoid. >> This is were I disagree, only a bit. It is true that using JNI >> *without >> care* can totally break the JVM, generate leaks, and bring down >> all >> the >> application. But if the JNI layer is developed carefully this >> should >> not >> be more risky than any of the other multiple native libraries that >> you >> use every day from Java without noticing. >> In addition JGIR is based on JNA (which is well maintained and >> stable) >> and doesn't include any "dangerous" native C code. >> > Well, you can call me old fashioned... > > But I don't think that any none critical task should be run out of > process of the production component. > > It is not GObject layer I am afraid of, it is what it eventually > runs. > > I believe that the data provided by libosinfo is not vital for our > runtime, no problem in fetching it before start. > > And again, I don't think data provider should be language > binding... this concept alone is making me scared, as instead of > using well defined data container people will start to use mesh of > components... I don't see how you can stabilize an application > this way. > > For the native library that you use "every day", I don't know what > you referring to, I've done lsof of engine and saw no exception... > which other native library we use these days? > >>> Having language binding for data access logic seems a bit strange >>> for me, as it should be easy to stabilize a schema either using >>> XML or a set of CSVs, for all to read and use. >>> >>> I do understand why JNI is to be used when OS specific features >>> are >>> to be used, for example unix domain sockets, shared memory, >>> security, but not for data access. >>> >>> The alternatives would be: >>> >>> 1. Run a python program at ovirt-engine start-up, use the python >>> API to fetch the data we need, and generate our own tiny XML of >>> the subset or even load it to the database. We can use the output >>> safely in Java. Disadvantage is that we need to stop/start >>> service >>> or have a "refresh" verb for reload after libosinfo update. >>> >>> 2. When data is requested, execute a python process with cmdline >>> arguments and get result from stdout of the process. The >>> disadvantage is the overhead of clone(). >>> >>> 3. Read the XML directly on runtime, disadvantage is if XML is >>> changed. >>> >>> 4. Use the libosinfo python API during build and distribute an >>> update to the information in versions of ovirt, disadvantage is >>> slow refresh of data, not sure how often this data is updated. >>> >>> 5. Provide a web service that has specific XML-RPC / RESTful API, >>> that can provide data out of the libosinfo, and publish it at >>> internet. Configure ovirt to use it, disadvantage is the need for >>> internet connection. >>> >>> 6. Configure cgi-bin locally at apache where ovirt is running, >>> which provide XML-RPC / RESTful API, and access it locally. >>> >>> In any case I would avoid exiting the JVM to interact with data >>> provider. >>> >>> Regards, >>> Alon. >>> >> >> -- >> Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, >> planta >> 3ºD, 28016 Madrid, Spain >> Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red >> Hat >> S.L. >> > _______________________________________________ > Engine-devel mailing list > Engine-devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel _______________________________________________ Engine-devel mailing list Engine-devel(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel

From: "Ewoud Kohl van Wijngaarden" <ewoud+ovirt(a)kohlvanwijngaarden.nl&gt; To: engine-devel(a)ovirt.org Sent: Thursday, October 11, 2012 12:38:12 PM Subject: Re: [Engine-devel] libosinfo integration On Thu, Oct 11, 2012 at 12:22:51PM +0200, Roy Golan wrote: > After considering the pros and cons and making poc's I've decided > to > take the direct XML and java pure implementation. > http://gerrit.ovirt.org/#/c/8472/ > http://gerrit.ovirt.org/#/c/8473/ Any particular reason I have no permission to view these?

From: "Roy Golan" <rgolan(a)redhat.com&gt; To: engine-devel(a)ovirt.org Sent: Thursday, October 11, 2012 12:45:58 PM Subject: Re: [Engine-devel] libosinfo integration On 10/11/2012 12:39 PM, Alon Bar-Lev wrote: > > ----- Original Message ----- >> From: "Ewoud Kohl van Wijngaarden" >> <ewoud+ovirt(a)kohlvanwijngaarden.nl&gt; >> To: engine-devel(a)ovirt.org >> Sent: Thursday, October 11, 2012 12:38:12 PM >> Subject: Re: [Engine-devel] libosinfo integration >> >> On Thu, Oct 11, 2012 at 12:22:51PM +0200, Roy Golan wrote: >>> After considering the pros and cons and making poc's I've decided >>> to >>> take the direct XML and java pure implementation. >>> http://gerrit.ovirt.org/#/c/8472/ >>> http://gerrit.ovirt.org/#/c/8473/ >> Any particular reason I have no permission to view these? > I cannot either... it must be a draft. > We should solve the drafts permission issues and make them public. Eedri do you know how to solve this?

I'll just add you manually meanwhile. > Alon. > _______________________________________________ > Engine-devel mailing list > Engine-devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel _______________________________________________ Engine-devel mailing list Engine-devel(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel