[urgent] Re: ovirt-engine build failure
Vojtech Szocs
vszocs at redhat.com
Wed Feb 26 11:25:50 UTC 2014
----- Original Message -----
> From: "Alon Bar-Lev" <alonbl at redhat.com>
> To: "Sandro Bonazzola" <sbonazzo at redhat.com>
> Cc: "Vojtech Szocs" <vszocs at redhat.com>, "Juan Hernandez" <jhernand at redhat.com>, "Kiril Nesenko"
> <knesenko at redhat.com>, "arch" <arch at ovirt.org>
> Sent: Wednesday, February 26, 2014 12:03:08 PM
> Subject: Re: [urgent] Re: ovirt-engine build failure
>
>
>
> ----- Original Message -----
> > From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> > To: "Vojtech Szocs" <vszocs at redhat.com>
> > Cc: "Juan Hernandez" <jhernand at redhat.com>, "Kiril Nesenko"
> > <knesenko at redhat.com>, "arch" <arch at ovirt.org>
> > Sent: Wednesday, February 26, 2014 12:58:29 PM
> > Subject: Re: [urgent] Re: ovirt-engine build failure
> >
> > Il 26/02/2014 11:51, Vojtech Szocs ha scritto:
> > >
> > >
> > > ----- Original Message -----
> > >> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> > >> To: "Tomas Jelinek" <tjelinek at redhat.com>
> > >> Cc: "Vojtech Szocs" <vszocs at redhat.com>, "Juan Hernandez"
> > >> <jhernand at redhat.com>, "Kiril Nesenko"
> > >> <knesenko at redhat.com>, "arch" <arch at ovirt.org>
> > >> Sent: Wednesday, February 26, 2014 10:22:22 AM
> > >> Subject: Re: [urgent] Re: ovirt-engine build failure
> > >>
> > >> Il 26/02/2014 09:06, Tomas Jelinek ha scritto:
> > >>> Hey,
> > >>>
> > >>> from this logs it seems that the root cause is:
> > >>> [ERROR] java.io.FileNotFoundException:
> > >>> /home/jenkins/workspace/manual-build-tarball/label/centos6-host/rpmbuild/BUILD/ovirt-engine-3.5.0/frontend/webadmin/modules/webadmin/target/tmp/ImageResourceGenerator4201639982561998434.png
> > >>> (Too many open files)
> > >>>
> > >>> e.g. the OS does not allow to open all the files the gwt compilation
> > >>> needs.
> > >>> Could you please try to increase the limit of max opened files on the
> > >>> host
> > >>> and try to rebuild?
> > >>
> > >> David is increasing max open files. We'll give it another run.
> > >> I really think that having to reduce localWorkers to 1 for having decent
> > >> build requirements on RAM and still hit too many opened files is really
> > >> bad.
> > >> Any chance of lowering build requirements in terms of RAM and opened
> > >> files?
> > >
> > > In general, GWT compiler generates/reads lots of intermediate files
> > > during
> > > its runtime. IIRC, this file amount is proportional to number of
> > > permutations
> > > being compiled, i.e. num_of_browsers x num_of_locales. This is why we
> > > don't
> > > hit "Too many open files" in small-scale builds (i.e. development) but we
> > > do
> > > hit it on large-scale builds (i.e. all_browsers x all_locales nightly
> > > build).
> > >
> > > But is increasing max open files for GWT compiler really that big of a
> > > deal?
> > > (i.e. we know that compiler needs to process lots of files per each
> > > permutation)
> >
> > No, it's not that big of a deal *for our packaging* but since we've people
> > on
> > Gentoo building engine
> > on their own systems, it may impact all of them...
>
> We can have clear note that limits should be modified to support locale
> build.
+1
Maybe this (or similar): "When building Engine for multiple/all locales, please check max open files limit to avoid errors during GWT compilation."
>
> We can run limit -n at Makefile and fail.
I'm not sure we can get good-enough/accurate "n" value since this depends on many different factors (including number of small images used in GWT application), so I'd just vote for clear note as suggested above.
>
> >
> > >
> > > Alternatively, we could implement "remote GWT worker grid" -> offload
> > > compilation
> > > for each specific (browser x locale) permutation to a remote machine,
> > > this
> > > would
> > > require some extra Java implementation though.
> > >
> > > I agree on excessive RAM usage, I'm currently experimenting with some
> > > compiler
> > > options, I'll share results when I have something.
> > >
> > > Another approach we could take (something I'll discuss with my peers) is
> > > the
> > > idea of eliminating locale vector entirely -> re-implement i18n support
> > > in
> > > UI
> > > in a way that for X browsers, there will be only X total permutations
> > > (i.e.
> > > not X x num_of_locales). This would reduce RAM usage && compilation time
> > > &&
> > > perhaps max files opened too.
> >
> > +1 on whatever can reduce build time and build requirements.
> >
> >
> > >
> > >>
> > >>
> > >>>
> > >>> ----- Original Message -----
> > >>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> > >>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> > >>>> Cc: "Juan Hernandez" <jhernand at redhat.com>, "Kiril Nesenko"
> > >>>> <knesenko at redhat.com>, "arch" <arch at ovirt.org>
> > >>>> Sent: Wednesday, February 26, 2014 8:37:07 AM
> > >>>> Subject: Re: [urgent] Re: ovirt-engine build failure
> > >>>>
> > >>>> Il 25/02/2014 16:52, Vojtech Szocs ha scritto:
> > >>>>>
> > >>>>>
> > >>>>> ----- Original Message -----
> > >>>>>> From: "Vojtech Szocs" <vszocs at redhat.com>
> > >>>>>> To: "Sandro Bonazzola" <sbonazzo at redhat.com>
> > >>>>>> Cc: "Juan Hernandez" <jhernand at redhat.com>, "Kiril Nesenko"
> > >>>>>> <knesenko at redhat.com>, "arch" <arch at ovirt.org>
> > >>>>>> Sent: Tuesday, February 25, 2014 1:05:35 PM
> > >>>>>> Subject: Re: [urgent] Re: ovirt-engine build failure
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> ----- Original Message -----
> > >>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> > >>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> > >>>>>>> Cc: "Juan Hernandez" <jhernand at redhat.com>, "arch"
> > >>>>>>> <arch at ovirt.org>,
> > >>>>>>> "Kiril
> > >>>>>>> Nesenko" <knesenko at redhat.com>
> > >>>>>>> Sent: Tuesday, February 25, 2014 9:02:11 AM
> > >>>>>>> Subject: Re: [urgent] Re: ovirt-engine build failure
> > >>>>>>>
> > >>>>>>> Il 25/02/2014 08:47, Sandro Bonazzola ha scritto:
> > >>>>>>>> Il 24/02/2014 18:28, Vojtech Szocs ha scritto:
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> ----- Original Message -----
> > >>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> > >>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> > >>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> > >>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> > >>>>>>>>>> "arch" <arch at ovirt.org>
> > >>>>>>>>>> Sent: Monday, February 24, 2014 9:08:09 AM
> > >>>>>>>>>> Subject: [urgent] Re: ovirt-engine build failure
> > >>>>>>>>>>
> > >>>>>>>>>> Top posting,
> > >>>>>>>>>>
> > >>>>>>>>>> Please give this task really urgent priority, as you can see:
> > >>>>>>>>>>
> > >>>>>>>>>> http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/
> > >>>>>>>>>> and
> > >>>>>>>>>> http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/
> > >>>>>>>>>>
> > >>>>>>>>>> are failing consistently so we don't have nightly builds.
> > >>>>>>>>>
> > >>>>>>>>> (CC'ing Einav)
> > >>>>>>>>>
> > >>>>>>>>> Attached TEST patch, please apply & rebuild ovirt-engine as
> > >>>>>>>>> usual.
> > >>>>>>>>>
> > >>>>>>>>> This patch should cause following output during GWT compilation:
> > >>>>>>>>>
> > >>>>>>>>> [ERROR] ######### Util.readURLAsBytes
> > >>>>>>>>> file:/home/vszocs/work/ovirt-engine/frontend/webadmin/modules/webadmin/target/generated-sources/gwt/org/ovirt/engine/ui/webadmin/images/search_button.png
> > >>>>>>>>>
> > >>>>>>>>> If IOException occurs, exception stacktrace will be printed right
> > >>>>>>>>> below
> > >>>>>>>>> that line.
> > >>>>>>>>>
> > >>>>>>>>> Note: please ignore the [ERROR] prefix, it's a technical issue,
> > >>>>>>>>> more
> > >>>>>>>>> details at
> > >>>>>>>>> https://www.mail-archive.com/engine-devel@ovirt.org/msg05735.html
> > >>>>>>>>>
> > >>>>>>>>> I just did a local upstream build with TEST patch applied, GWT
> > >>>>>>>>> compilation
> > >>>>>>>>> passed OK.
> > >>>>>>>>
> > >>>>>>>> build job running with your patch here:
> > >>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/235/
> > >>>>>>>> waiting for a free executor right now.
> > >>>>>>>> Let's see how it goes.
> > >>>>>>>
> > >>>>>>> Failed on CentOS:
> > >>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/235/label=centos6-host/consoleFull
> > >>>>>>
> > >>>>>> Hm, in that ^^ build, following exceptions occured:
> > >>>>>>
> > >>>>>> 1.
> > >>>>>> [INFO] java.lang.NullPointerException
> > >>>>>> [INFO] at
> > >>>>>> com.google.gwt.dev.util.Util.computeStrongName(Util.java:172)
> > >>>>>>
> > >>>>>> 2.
> > >>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed
> > >>>>>> for
> > >>>>>> downDisabledImage
> > >>>>>> [INFO] at
> > >>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475
> > >>>>>>
> > >>>>>> 1. is caused by Util#readURLAsBytes but I don't see any IOException
> > >>>>>> in
> > >>>>>> the
> > >>>>>> logs
> > >>>>>> 2. is just follow-up consequence of 1. (i.e. image metadata missing
> > >>>>>> in
> > >>>>>> GWT
> > >>>>>> image cache)
> > >>>>>>
> > >>>>>> Now I noticed I missed one possible IOException inside
> > >>>>>> Util#readURLConnectionAsBytes ...
> > >>>>>>
> > >>>>>> Long story short, I'll send updated TEST patch soon, apologies for
> > >>>>>> the
> > >>>>>> inconvenience.
> > >>>>>
> > >>>>> TEST-v2 patch attached. Please apply & rebuild ovirt-engine as usual.
> > >>>>
> > >>>> Done, here is the console output:
> > >>>> http://jenkins.ovirt.org/job/manual-build-tarball/label=centos6-host/239/consoleFull
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>>
> > >>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Il 20/02/2014 13:34, Vojtech Szocs ha scritto:
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> ----- Original Message -----
> > >>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> > >>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> > >>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> > >>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> > >>>>>>>>>>>> "arch" <arch at ovirt.org>
> > >>>>>>>>>>>> Sent: Thursday, February 20, 2014 10:02:00 AM
> > >>>>>>>>>>>> Subject: Re: ovirt-engine build failure
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Il 19/02/2014 16:51, Vojtech Szocs ha scritto:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> ----- Original Message -----
> > >>>>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> > >>>>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> > >>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> > >>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko"
> > >>>>>>>>>>>>>> <knesenko at redhat.com>,
> > >>>>>>>>>>>>>> "arch" <arch at ovirt.org>
> > >>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 2:43:44 PM
> > >>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Il 19/02/2014 14:41, Vojtech Szocs ha scritto:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> ----- Original Message -----
> > >>>>>>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> > >>>>>>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> > >>>>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> > >>>>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko"
> > >>>>>>>>>>>>>>>> <knesenko at redhat.com>,
> > >>>>>>>>>>>>>>>> "arch" <arch at ovirt.org>
> > >>>>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 2:23:16 PM
> > >>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Il 19/02/2014 14:19, Vojtech Szocs ha scritto:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> ----- Original Message -----
> > >>>>>>>>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> > >>>>>>>>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> > >>>>>>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> > >>>>>>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko"
> > >>>>>>>>>>>>>>>>>> <knesenko at redhat.com>,
> > >>>>>>>>>>>>>>>>>> "arch" <arch at ovirt.org>
> > >>>>>>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 1:18:28 PM
> > >>>>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto:
> > >>>>>>>>>>>>>>>>>>> Hi, I checked GWT sources:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect
> > >>>>>>>>>>>>>>>>>>>> ever
> > >>>>>>>>>>>>>>>>>>>> computed
> > >>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>> removeBrickCommitRequired
> > >>>>>>>>>>>>>>>>>>>> [INFO] at
> > >>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> This happened because internal image cache for bundled
> > >>>>>>>>>>>>>>>>>>> (composite
> > >>>>>>>>>>>>>>>>>>> PNG)
> > >>>>>>>>>>>>>>>>>>> images was corrput.
> > >>>>>>>>>>>>>>>>>>> It seems to be caused by disk I/O error while reading
> > >>>>>>>>>>>>>>>>>>> file
> > >>>>>>>>>>>>>>>>>>> images/glusterAsyncTasks/remove_brick_commit_required.png
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException
> > >>>>>>>>>>>>>>>>>>>> [INFO] at
> > >>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> This happened because image data was null, again
> > >>>>>>>>>>>>>>>>>>> possible
> > >>>>>>>>>>>>>>>>>>> disk
> > >>>>>>>>>>>>>>>>>>> I/O
> > >>>>>>>>>>>>>>>>>>> error
> > >>>>>>>>>>>>>>>>>>> while reading image file.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Can we enable "TRACE" logging during GWT compilation?
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> -Dgwt.logLevel=TRACE
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Building master right now with your suggestions here:
> > >>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Looking at
> > >>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host/console
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> So far I see no errors.. There should be total 5 browsers
> > >>>>>>>>>>>>>>>>> x
> > >>>>>>>>>>>>>>>>> 8
> > >>>>>>>>>>>>>>>>> locales
> > >>>>>>>>>>>>>>>>> =
> > >>>>>>>>>>>>>>>>> 40
> > >>>>>>>>>>>>>>>>> GWT permutations.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> So no changes with and without
> > >>>>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera.
> > >>>>>>>>>>>>>>>> We've still 40 permutations. Maybe you defined
> > >>>>>>>>>>>>>>>> userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the
> > >>>>>>>>>>>>>>>> POM
> > >>>>>>>>>>>>>>>> files?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> You are right. I checked frontend/webadmin/modules/pom.xml
> > >>>>>>>>>>>>>>> ->
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>> "gwt.userAgent"
> > >>>>>>>>>>>>>>> property is copied into "locale" GWT property in
> > >>>>>>>>>>>>>>> {WebAdmin,UserPortal}.gwt.xml
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Ok, so I'll remove
> > >>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera
> > >>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>> build
> > >>>>>>>>>>>>>> command.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Agreed.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> This:
> > >>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/console
> > >>>>>>>>>>>>>>>> failed, you can look at it, it has -Dgwt.logLevel=TRACE
> > >>>>>>>>>>>>>>>> -X.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hm, it contains:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> [DEBUG] (f) logLevel = INFO
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Somehow the TRACE log level for GWT wasn't applied.. not
> > >>>>>>>>>>>>>>> sure
> > >>>>>>>>>>>>>>> why.
> > >>>>>>>>>>>>>>> Let
> > >>>>>>>>>>>>>>> me
> > >>>>>>>>>>>>>>> check
> > >>>>>>>>>>>>>>> this on local build.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Ok, let me know if I've to change command line.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> When it's merged, setting "gwt.logLevel" will work (as you
> > >>>>>>>>>>>>> did
> > >>>>>>>>>>>>> before)
> > >>>>>>>>>>>>> and we'll be able to see more details in the build log.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Here is the new build with your patch merged:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/consoleFull
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks. I did some more investigation on this.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Again it's NullPointerException when rebinding
> > >>>>>>>>>>> ApplicationResources,
> > >>>>>>>>>>> but
> > >>>>>>>>>>> this time for different image file:
> > >>>>>>>>>>> ApplicationResources.wrenchImage
> > >>>>>>>>>>> ("images/wrench.png").
> > >>>>>>>>>>>
> > >>>>>>>>>>> Looking at stacktrace & GWT sources, the problematic code seems
> > >>>>>>>>>>> to
> > >>>>>>>>>>> be
> > >>>>>>>>>>> (AbstractResourceContext:74):
> > >>>>>>>>>>>
> > >>>>>>>>>>> byte[] bytes = Util.readURLAsBytes(resource);
> > >>>>>>>>>>>
> > >>>>>>>>>>> where "resource" is (ImageResourceGenerator:311):
> > >>>>>>>>>>>
> > >>>>>>>>>>> localized.getUrl(); // LocalizedImage localized
> > >>>>>>>>>>>
> > >>>>>>>>>>> Note: in oVirt UI we don't use localized images (i.e.
> > >>>>>>>>>>> "wrench_fr_FR.png")
> > >>>>>>>>>>> so "localized" in this case points to "wrench.png".
> > >>>>>>>>>>>
> > >>>>>>>>>>> So the problem seems to be reading this file:
> > >>>>>>>>>>>
> > >>>>>>>>>>> frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png
> > >>>>>>>>>>>
> > >>>>>>>>>>> Since we're having many GWT permutations, this file ^^ (along
> > >>>>>>>>>>> with
> > >>>>>>>>>>> similar
> > >>>>>>>>>>> ones) is attempted to be read multiple times *from the same
> > >>>>>>>>>>> (single)
> > >>>>>>>>>>> GWT
> > >>>>>>>>>>> compiler Java process, before actual permutation worker Java
> > >>>>>>>>>>> sub-processes
> > >>>>>>>>>>> are spawned*. (More precisely, above NullPointerException
> > >>>>>>>>>>> happens
> > >>>>>>>>>>> at
> > >>>>>>>>>>> deferred binding phase which precedes actual GWT compilation
> > >>>>>>>>>>> per
> > >>>>>>>>>>> each
> > >>>>>>>>>>> permutation.)
> > >>>>>>>>>>>
> > >>>>>>>>>>> I think the root cause is Util.readURLAsBytes method used
> > >>>>>>>>>>> internally
> > >>>>>>>>>>> by
> > >>>>>>>>>>> GWT
> > >>>>>>>>>>> compiler:
> > >>>>>>>>>>>
> > >>>>>>>>>>> public static byte[] readURLAsBytes(URL url) {
> > >>>>>>>>>>> try {
> > >>>>>>>>>>> URLConnection conn = url.openConnection();
> > >>>>>>>>>>> conn.setUseCaches(false);
> > >>>>>>>>>>> return readURLConnectionAsBytes(conn);
> > >>>>>>>>>>> } catch (IOException e) {
> > >>>>>>>>>>> return null; // This causes NullPointerException, but
> > >>>>>>>>>>> this
> > >>>>>>>>>>> exception
> > >>>>>>>>>>> isn't logged (wtf, GWT!)
> > >>>>>>>>>>> }
> > >>>>>>>>>>> }
> > >>>>>>>>>>>
> > >>>>>>>>>>> To summarize, I'll need to write another patch which replaces
> > >>>>>>>>>>> Util.readURLAsBytes implementation via AOP and prints
> > >>>>>>>>>>> IOException
> > >>>>>>>>>>> details.
> > >>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> #
> > >>>>>>>>>>>>>>>>>> CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml
> > >>>>>>>>>>>>>>>>>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M
> > >>>>>>>>>>>>>>>>>> -XX:MaxPermSize=1024m
> > >>>>>>>>>>>>>>>>>> -Dgwt.compiler.localWorkers=1"
> > >>>>>>>>>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}"
> > >>>>>>>>>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D
> > >>>>>>>>>>>>>>>>>> "ovirt_build_extra_flags
> > >>>>>>>>>>>>>>>>>> -gs
> > >>>>>>>>>>>>>>>>>> ${CI_MAVEN_SETTINGS}
> > >>>>>>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera
> > >>>>>>>>>>>>>>>>>> -Dgwt.logLevel=TRACE -X" --rebuild
> > >>>>>>>>>>>>>>>>>> rpmbuild/SRPMS/*.src.rpm
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Let me know if you need anything else.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Vojtech
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> ----- Original Message -----
> > >>>>>>>>>>>>>>>>>>>> From: "Vojtech Szocs" <vszocs at redhat.com>
> > >>>>>>>>>>>>>>>>>>>> To: "Sandro Bonazzola" <sbonazzo at redhat.com>
> > >>>>>>>>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> > >>>>>>>>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko"
> > >>>>>>>>>>>>>>>>>>>> <knesenko at redhat.com>,
> > >>>>>>>>>>>>>>>>>>>> "arch" <arch at ovirt.org>
> > >>>>>>>>>>>>>>>>>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM
> > >>>>>>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> In your log I see following error:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> ...
> > >>>>>>>>>>>>>>>>>>>> [INFO] Computing all possible rebind results for
> > >>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> > >>>>>>>>>>>>>>>>>>>> [INFO] Rebinding
> > >>>>>>>>>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
> > >>>>>>>>>>>>>>>>>>>> [INFO] Invoking generator
> > >>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
> > >>>>>>>>>>>>>>>>>>>> [INFO] [ERROR] Generator
> > >>>>>>>>>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
> > >>>>>>>>>>>>>>>>>>>> threw
> > >>>>>>>>>>>>>>>>>>>> an exception while rebinding
> > >>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> > >>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect
> > >>>>>>>>>>>>>>>>>>>> ever
> > >>>>>>>>>>>>>>>>>>>> computed
> > >>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>> removeBrickCommitRequired
> > >>>>>>>>>>>>>>>>>>>> [INFO] at
> > >>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
> > >>>>>>>>>>>>>>>>>>>> ...
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> I checked again logs at
> > >>>>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host/consoleFull
> > >>>>>>>>>>>>>>>>>>>> and indeed there is a similar error:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> ...
> > >>>>>>>>>>>>>>>>>>>> [INFO] Computing all possible rebind results for
> > >>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> > >>>>>>>>>>>>>>>>>>>> [INFO] Rebinding
> > >>>>>>>>>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
> > >>>>>>>>>>>>>>>>>>>> [INFO] Invoking generator
> > >>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
> > >>>>>>>>>>>>>>>>>>>> [INFO] [ERROR] Generator
> > >>>>>>>>>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
> > >>>>>>>>>>>>>>>>>>>> threw
> > >>>>>>>>>>>>>>>>>>>> an exception while rebinding
> > >>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> > >>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException
> > >>>>>>>>>>>>>>>>>>>> [INFO] at
> > >>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
> > >>>>>>>>>>>>>>>>>>>> ...
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> NullPointerException when processing
> > >>>>>>>>>>>>>>>>>>>> ApplicationResources
> > >>>>>>>>>>>>>>>>>>>> but
> > >>>>>>>>>>>>>>>>>>>> occuring
> > >>>>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>> different places (ImageResourceGenerator vs.
> > >>>>>>>>>>>>>>>>>>>> InlineResourceContext)
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Now I understand why Maven build fails without any
> > >>>>>>>>>>>>>>>>>>>> details:
> > >>>>>>>>>>>>>>>>>>>> - GWT deferred binding fails for ApplicationResources
> > >>>>>>>>>>>>>>>>>>>> (i.e.
> > >>>>>>>>>>>>>>>>>>>> uncaught
> > >>>>>>>>>>>>>>>>>>>> NPE)
> > >>>>>>>>>>>>>>>>>>>> - GWT compilation proceeds, fails to link with
> > >>>>>>>>>>>>>>>>>>>> ApplicationResources
> > >>>>>>>>>>>>>>>>>>>> - GWT compilation task fails with non-OK status code
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> So the issue here is GWT deferred binding generator
> > >>>>>>>>>>>>>>>>>>>> (InlineClientBundleGenerator) failure during GWT
> > >>>>>>>>>>>>>>>>>>>> compilation.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> GWT compiler writes extra Java files during deferred
> > >>>>>>>>>>>>>>>>>>>> binding
> > >>>>>>>>>>>>>>>>>>>> phase, so this could point to issue with local
> > >>>>>>>>>>>>>>>>>>>> environment..
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> I will try to investigate this and look at GWT source
> > >>>>>>>>>>>>>>>>>>>> code.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>>>>> Sandro Bonazzola
> > >>>>>>>>>>>>>>>>>> Better technology. Faster innovation. Powered by
> > >>>>>>>>>>>>>>>>>> community
> > >>>>>>>>>>>>>>>>>> collaboration.
> > >>>>>>>>>>>>>>>>>> See how it works at redhat.com
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>>> Sandro Bonazzola
> > >>>>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community
> > >>>>>>>>>>>>>>>> collaboration.
> > >>>>>>>>>>>>>>>> See how it works at redhat.com
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>> Sandro Bonazzola
> > >>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community
> > >>>>>>>>>>>>>> collaboration.
> > >>>>>>>>>>>>>> See how it works at redhat.com
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> --
> > >>>>>>>>>>>> Sandro Bonazzola
> > >>>>>>>>>>>> Better technology. Faster innovation. Powered by community
> > >>>>>>>>>>>> collaboration.
> > >>>>>>>>>>>> See how it works at redhat.com
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> --
> > >>>>>>>>>> Sandro Bonazzola
> > >>>>>>>>>> Better technology. Faster innovation. Powered by community
> > >>>>>>>>>> collaboration.
> > >>>>>>>>>> See how it works at redhat.com
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> --
> > >>>>>>> Sandro Bonazzola
> > >>>>>>> Better technology. Faster innovation. Powered by community
> > >>>>>>> collaboration.
> > >>>>>>> See how it works at redhat.com
> > >>>>>>>
> > >>>>>> _______________________________________________
> > >>>>>> Arch mailing list
> > >>>>>> Arch at ovirt.org
> > >>>>>> http://lists.ovirt.org/mailman/listinfo/arch
> > >>>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Sandro Bonazzola
> > >>>> Better technology. Faster innovation. Powered by community
> > >>>> collaboration.
> > >>>> See how it works at redhat.com
> > >>>> _______________________________________________
> > >>>> Arch mailing list
> > >>>> Arch at ovirt.org
> > >>>> http://lists.ovirt.org/mailman/listinfo/arch
> > >>>>
> > >>
> > >>
> > >> --
> > >> Sandro Bonazzola
> > >> Better technology. Faster innovation. Powered by community
> > >> collaboration.
> > >> See how it works at redhat.com
> > >>
> >
> >
> > --
> > Sandro Bonazzola
> > Better technology. Faster innovation. Powered by community collaboration.
> > See how it works at redhat.com
> > _______________________________________________
> > Arch mailing list
> > Arch at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/arch
> >
>
More information about the Arch
mailing list