[urgent] Re: ovirt-engine build failure
Vojtech Szocs
vszocs at redhat.com
Wed Feb 26 15:40:55 UTC 2014
(top posting)
I did some preliminary tests on my local machine, invoking "mvn clean install -Pgwt-admin" command to build (only) WebAdmin from source.
Default permutation settings: all (5) browsers, one locale = 5 total permutations to compile
Tested two different strategies (machine with 16G RAM / 8G swap, quad-core CPU):
A, localWorkers=X (default external-JVM worker factory)
-Dgwt.compiler.localWorkers=X
B, localWorkers=X + maxThreads=X (thread-based worker factory)
-Dgwt.compiler.localWorkers=X -Dgwt.jjs.permutationWorkerFactory=com.google.gwt.dev.ThreadedPermutationWorkerFactory -Dgwt.jjs.maxThreads=X
A -> this is what we use today
B -> this is possible alternative
X=4 [5 perms]
A, initial 2.6 GiB, peak 13.5 GiB
Total time: 5:25.851s
B, initial 2.6 GiB, peak 13.1 GiB
Total time: 5:11.238s
X=1 [5 perms]
A, initial 2.6 GiB, peak 7.1 GiB
Total time: 7:00.601s
B, initial 2.6 GiB, peak 7.4 GiB
Total time: 7:07.458s
Now switched to permutation settings: all (5) browsers, 3 locales = 15 total permutations to compile
(Used swapoff/swapon to force swap clear after each test run.)
X=4 [15 perms]
A, initial 2.6 GiB, peak physical max + swap ~4.3 GiB
Total time: 11:39.086s
B, initial 2.6 GiB, peak physical max + swap ~3.5 GiB
Total time: 10:59.966s
X=1 [15 perms]
A, initial 2.6 GiB, peak 7.7 GiB
Total time: 17:39.103s
B, initial 2.6 GiB. peak 7.9 GiB
Total time: 17:16.314s
For X=1, both strategies (A & B) have very similar results.
For X=4 (more than one worker), thread-based strategy seems slightly better, but still we have huge RAM consumption + max opened files quite high.
I think the only real solution is to cut off (eliminate) locale permutation vector to reduce amount of GWT permutations. It's on my task list.
Vojtech
----- Original Message -----
> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> To: "Vojtech Szocs" <vszocs at redhat.com>
> Cc: "Tomas Jelinek" <tjelinek at redhat.com>, "Juan Hernandez" <jhernand at redhat.com>, "Kiril Nesenko"
> <knesenko at redhat.com>, "arch" <arch at ovirt.org>
> Sent: Wednesday, February 26, 2014 11:58:29 AM
> Subject: Re: [urgent] Re: ovirt-engine build failure
>
> Il 26/02/2014 11:51, Vojtech Szocs ha scritto:
> >
> >
> > ----- Original Message -----
> >> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >> To: "Tomas Jelinek" <tjelinek at redhat.com>
> >> Cc: "Vojtech Szocs" <vszocs at redhat.com>, "Juan Hernandez"
> >> <jhernand at redhat.com>, "Kiril Nesenko"
> >> <knesenko at redhat.com>, "arch" <arch at ovirt.org>
> >> Sent: Wednesday, February 26, 2014 10:22:22 AM
> >> Subject: Re: [urgent] Re: ovirt-engine build failure
> >>
> >> Il 26/02/2014 09:06, Tomas Jelinek ha scritto:
> >>> Hey,
> >>>
> >>> from this logs it seems that the root cause is:
> >>> [ERROR] java.io.FileNotFoundException:
> >>> /home/jenkins/workspace/manual-build-tarball/label/centos6-host/rpmbuild/BUILD/ovirt-engine-3.5.0/frontend/webadmin/modules/webadmin/target/tmp/ImageResourceGenerator4201639982561998434.png
> >>> (Too many open files)
> >>>
> >>> e.g. the OS does not allow to open all the files the gwt compilation
> >>> needs.
> >>> Could you please try to increase the limit of max opened files on the
> >>> host
> >>> and try to rebuild?
> >>
> >> David is increasing max open files. We'll give it another run.
> >> I really think that having to reduce localWorkers to 1 for having decent
> >> build requirements on RAM and still hit too many opened files is really
> >> bad.
> >> Any chance of lowering build requirements in terms of RAM and opened
> >> files?
> >
> > In general, GWT compiler generates/reads lots of intermediate files during
> > its runtime. IIRC, this file amount is proportional to number of
> > permutations
> > being compiled, i.e. num_of_browsers x num_of_locales. This is why we don't
> > hit "Too many open files" in small-scale builds (i.e. development) but we
> > do
> > hit it on large-scale builds (i.e. all_browsers x all_locales nightly
> > build).
> >
> > But is increasing max open files for GWT compiler really that big of a
> > deal?
> > (i.e. we know that compiler needs to process lots of files per each
> > permutation)
>
> No, it's not that big of a deal *for our packaging* but since we've people on
> Gentoo building engine
> on their own systems, it may impact all of them...
>
> >
> > Alternatively, we could implement "remote GWT worker grid" -> offload
> > compilation
> > for each specific (browser x locale) permutation to a remote machine, this
> > would
> > require some extra Java implementation though.
> >
> > I agree on excessive RAM usage, I'm currently experimenting with some
> > compiler
> > options, I'll share results when I have something.
> >
> > Another approach we could take (something I'll discuss with my peers) is
> > the
> > idea of eliminating locale vector entirely -> re-implement i18n support in
> > UI
> > in a way that for X browsers, there will be only X total permutations (i.e.
> > not X x num_of_locales). This would reduce RAM usage && compilation time &&
> > perhaps max files opened too.
>
> +1 on whatever can reduce build time and build requirements.
>
>
> >
> >>
> >>
> >>>
> >>> ----- Original Message -----
> >>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>> Cc: "Juan Hernandez" <jhernand at redhat.com>, "Kiril Nesenko"
> >>>> <knesenko at redhat.com>, "arch" <arch at ovirt.org>
> >>>> Sent: Wednesday, February 26, 2014 8:37:07 AM
> >>>> Subject: Re: [urgent] Re: ovirt-engine build failure
> >>>>
> >>>> Il 25/02/2014 16:52, Vojtech Szocs ha scritto:
> >>>>>
> >>>>>
> >>>>> ----- Original Message -----
> >>>>>> From: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>> To: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>> Cc: "Juan Hernandez" <jhernand at redhat.com>, "Kiril Nesenko"
> >>>>>> <knesenko at redhat.com>, "arch" <arch at ovirt.org>
> >>>>>> Sent: Tuesday, February 25, 2014 1:05:35 PM
> >>>>>> Subject: Re: [urgent] Re: ovirt-engine build failure
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ----- Original Message -----
> >>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>> Cc: "Juan Hernandez" <jhernand at redhat.com>, "arch" <arch at ovirt.org>,
> >>>>>>> "Kiril
> >>>>>>> Nesenko" <knesenko at redhat.com>
> >>>>>>> Sent: Tuesday, February 25, 2014 9:02:11 AM
> >>>>>>> Subject: Re: [urgent] Re: ovirt-engine build failure
> >>>>>>>
> >>>>>>> Il 25/02/2014 08:47, Sandro Bonazzola ha scritto:
> >>>>>>>> Il 24/02/2014 18:28, Vojtech Szocs ha scritto:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ----- Original Message -----
> >>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> >>>>>>>>>> "arch" <arch at ovirt.org>
> >>>>>>>>>> Sent: Monday, February 24, 2014 9:08:09 AM
> >>>>>>>>>> Subject: [urgent] Re: ovirt-engine build failure
> >>>>>>>>>>
> >>>>>>>>>> Top posting,
> >>>>>>>>>>
> >>>>>>>>>> Please give this task really urgent priority, as you can see:
> >>>>>>>>>>
> >>>>>>>>>> http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/
> >>>>>>>>>> and
> >>>>>>>>>> http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/
> >>>>>>>>>>
> >>>>>>>>>> are failing consistently so we don't have nightly builds.
> >>>>>>>>>
> >>>>>>>>> (CC'ing Einav)
> >>>>>>>>>
> >>>>>>>>> Attached TEST patch, please apply & rebuild ovirt-engine as usual.
> >>>>>>>>>
> >>>>>>>>> This patch should cause following output during GWT compilation:
> >>>>>>>>>
> >>>>>>>>> [ERROR] ######### Util.readURLAsBytes
> >>>>>>>>> file:/home/vszocs/work/ovirt-engine/frontend/webadmin/modules/webadmin/target/generated-sources/gwt/org/ovirt/engine/ui/webadmin/images/search_button.png
> >>>>>>>>>
> >>>>>>>>> If IOException occurs, exception stacktrace will be printed right
> >>>>>>>>> below
> >>>>>>>>> that line.
> >>>>>>>>>
> >>>>>>>>> Note: please ignore the [ERROR] prefix, it's a technical issue,
> >>>>>>>>> more
> >>>>>>>>> details at
> >>>>>>>>> https://www.mail-archive.com/engine-devel@ovirt.org/msg05735.html
> >>>>>>>>>
> >>>>>>>>> I just did a local upstream build with TEST patch applied, GWT
> >>>>>>>>> compilation
> >>>>>>>>> passed OK.
> >>>>>>>>
> >>>>>>>> build job running with your patch here:
> >>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/235/
> >>>>>>>> waiting for a free executor right now.
> >>>>>>>> Let's see how it goes.
> >>>>>>>
> >>>>>>> Failed on CentOS:
> >>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/235/label=centos6-host/consoleFull
> >>>>>>
> >>>>>> Hm, in that ^^ build, following exceptions occured:
> >>>>>>
> >>>>>> 1.
> >>>>>> [INFO] java.lang.NullPointerException
> >>>>>> [INFO] at
> >>>>>> com.google.gwt.dev.util.Util.computeStrongName(Util.java:172)
> >>>>>>
> >>>>>> 2.
> >>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed for
> >>>>>> downDisabledImage
> >>>>>> [INFO] at
> >>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475
> >>>>>>
> >>>>>> 1. is caused by Util#readURLAsBytes but I don't see any IOException in
> >>>>>> the
> >>>>>> logs
> >>>>>> 2. is just follow-up consequence of 1. (i.e. image metadata missing in
> >>>>>> GWT
> >>>>>> image cache)
> >>>>>>
> >>>>>> Now I noticed I missed one possible IOException inside
> >>>>>> Util#readURLConnectionAsBytes ...
> >>>>>>
> >>>>>> Long story short, I'll send updated TEST patch soon, apologies for the
> >>>>>> inconvenience.
> >>>>>
> >>>>> TEST-v2 patch attached. Please apply & rebuild ovirt-engine as usual.
> >>>>
> >>>> Done, here is the console output:
> >>>> http://jenkins.ovirt.org/job/manual-build-tarball/label=centos6-host/239/consoleFull
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Il 20/02/2014 13:34, Vojtech Szocs ha scritto:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> >>>>>>>>>>>> "arch" <arch at ovirt.org>
> >>>>>>>>>>>> Sent: Thursday, February 20, 2014 10:02:00 AM
> >>>>>>>>>>>> Subject: Re: ovirt-engine build failure
> >>>>>>>>>>>>
> >>>>>>>>>>>> Il 19/02/2014 16:51, Vojtech Szocs ha scritto:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> >>>>>>>>>>>>>> "arch" <arch at ovirt.org>
> >>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 2:43:44 PM
> >>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Il 19/02/2014 14:41, Vojtech Szocs ha scritto:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko"
> >>>>>>>>>>>>>>>> <knesenko at redhat.com>,
> >>>>>>>>>>>>>>>> "arch" <arch at ovirt.org>
> >>>>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 2:23:16 PM
> >>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Il 19/02/2014 14:19, Vojtech Szocs ha scritto:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>>>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>>>>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko"
> >>>>>>>>>>>>>>>>>> <knesenko at redhat.com>,
> >>>>>>>>>>>>>>>>>> "arch" <arch at ovirt.org>
> >>>>>>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 1:18:28 PM
> >>>>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto:
> >>>>>>>>>>>>>>>>>>> Hi, I checked GWT sources:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever
> >>>>>>>>>>>>>>>>>>>> computed
> >>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>> removeBrickCommitRequired
> >>>>>>>>>>>>>>>>>>>> [INFO] at
> >>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> This happened because internal image cache for bundled
> >>>>>>>>>>>>>>>>>>> (composite
> >>>>>>>>>>>>>>>>>>> PNG)
> >>>>>>>>>>>>>>>>>>> images was corrput.
> >>>>>>>>>>>>>>>>>>> It seems to be caused by disk I/O error while reading
> >>>>>>>>>>>>>>>>>>> file
> >>>>>>>>>>>>>>>>>>> images/glusterAsyncTasks/remove_brick_commit_required.png
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException
> >>>>>>>>>>>>>>>>>>>> [INFO] at
> >>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> This happened because image data was null, again possible
> >>>>>>>>>>>>>>>>>>> disk
> >>>>>>>>>>>>>>>>>>> I/O
> >>>>>>>>>>>>>>>>>>> error
> >>>>>>>>>>>>>>>>>>> while reading image file.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Can we enable "TRACE" logging during GWT compilation?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -Dgwt.logLevel=TRACE
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Building master right now with your suggestions here:
> >>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Looking at
> >>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host/console
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> So far I see no errors.. There should be total 5 browsers x
> >>>>>>>>>>>>>>>>> 8
> >>>>>>>>>>>>>>>>> locales
> >>>>>>>>>>>>>>>>> =
> >>>>>>>>>>>>>>>>> 40
> >>>>>>>>>>>>>>>>> GWT permutations.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> So no changes with and without
> >>>>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera.
> >>>>>>>>>>>>>>>> We've still 40 permutations. Maybe you defined
> >>>>>>>>>>>>>>>> userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the POM
> >>>>>>>>>>>>>>>> files?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> You are right. I checked frontend/webadmin/modules/pom.xml ->
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>> "gwt.userAgent"
> >>>>>>>>>>>>>>> property is copied into "locale" GWT property in
> >>>>>>>>>>>>>>> {WebAdmin,UserPortal}.gwt.xml
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ok, so I'll remove
> >>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera
> >>>>>>>>>>>>>> from
> >>>>>>>>>>>>>> build
> >>>>>>>>>>>>>> command.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Agreed.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> This:
> >>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/console
> >>>>>>>>>>>>>>>> failed, you can look at it, it has -Dgwt.logLevel=TRACE -X.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hm, it contains:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [DEBUG] (f) logLevel = INFO
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Somehow the TRACE log level for GWT wasn't applied.. not sure
> >>>>>>>>>>>>>>> why.
> >>>>>>>>>>>>>>> Let
> >>>>>>>>>>>>>>> me
> >>>>>>>>>>>>>>> check
> >>>>>>>>>>>>>>> this on local build.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ok, let me know if I've to change command line.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> When it's merged, setting "gwt.logLevel" will work (as you did
> >>>>>>>>>>>>> before)
> >>>>>>>>>>>>> and we'll be able to see more details in the build log.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Here is the new build with your patch merged:
> >>>>>>>>>>>>
> >>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/consoleFull
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks. I did some more investigation on this.
> >>>>>>>>>>>
> >>>>>>>>>>> Again it's NullPointerException when rebinding
> >>>>>>>>>>> ApplicationResources,
> >>>>>>>>>>> but
> >>>>>>>>>>> this time for different image file:
> >>>>>>>>>>> ApplicationResources.wrenchImage
> >>>>>>>>>>> ("images/wrench.png").
> >>>>>>>>>>>
> >>>>>>>>>>> Looking at stacktrace & GWT sources, the problematic code seems
> >>>>>>>>>>> to
> >>>>>>>>>>> be
> >>>>>>>>>>> (AbstractResourceContext:74):
> >>>>>>>>>>>
> >>>>>>>>>>> byte[] bytes = Util.readURLAsBytes(resource);
> >>>>>>>>>>>
> >>>>>>>>>>> where "resource" is (ImageResourceGenerator:311):
> >>>>>>>>>>>
> >>>>>>>>>>> localized.getUrl(); // LocalizedImage localized
> >>>>>>>>>>>
> >>>>>>>>>>> Note: in oVirt UI we don't use localized images (i.e.
> >>>>>>>>>>> "wrench_fr_FR.png")
> >>>>>>>>>>> so "localized" in this case points to "wrench.png".
> >>>>>>>>>>>
> >>>>>>>>>>> So the problem seems to be reading this file:
> >>>>>>>>>>>
> >>>>>>>>>>> frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png
> >>>>>>>>>>>
> >>>>>>>>>>> Since we're having many GWT permutations, this file ^^ (along
> >>>>>>>>>>> with
> >>>>>>>>>>> similar
> >>>>>>>>>>> ones) is attempted to be read multiple times *from the same
> >>>>>>>>>>> (single)
> >>>>>>>>>>> GWT
> >>>>>>>>>>> compiler Java process, before actual permutation worker Java
> >>>>>>>>>>> sub-processes
> >>>>>>>>>>> are spawned*. (More precisely, above NullPointerException happens
> >>>>>>>>>>> at
> >>>>>>>>>>> deferred binding phase which precedes actual GWT compilation per
> >>>>>>>>>>> each
> >>>>>>>>>>> permutation.)
> >>>>>>>>>>>
> >>>>>>>>>>> I think the root cause is Util.readURLAsBytes method used
> >>>>>>>>>>> internally
> >>>>>>>>>>> by
> >>>>>>>>>>> GWT
> >>>>>>>>>>> compiler:
> >>>>>>>>>>>
> >>>>>>>>>>> public static byte[] readURLAsBytes(URL url) {
> >>>>>>>>>>> try {
> >>>>>>>>>>> URLConnection conn = url.openConnection();
> >>>>>>>>>>> conn.setUseCaches(false);
> >>>>>>>>>>> return readURLConnectionAsBytes(conn);
> >>>>>>>>>>> } catch (IOException e) {
> >>>>>>>>>>> return null; // This causes NullPointerException, but this
> >>>>>>>>>>> exception
> >>>>>>>>>>> isn't logged (wtf, GWT!)
> >>>>>>>>>>> }
> >>>>>>>>>>> }
> >>>>>>>>>>>
> >>>>>>>>>>> To summarize, I'll need to write another patch which replaces
> >>>>>>>>>>> Util.readURLAsBytes implementation via AOP and prints IOException
> >>>>>>>>>>> details.
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> #
> >>>>>>>>>>>>>>>>>> CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml
> >>>>>>>>>>>>>>>>>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M
> >>>>>>>>>>>>>>>>>> -XX:MaxPermSize=1024m
> >>>>>>>>>>>>>>>>>> -Dgwt.compiler.localWorkers=1"
> >>>>>>>>>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}"
> >>>>>>>>>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D
> >>>>>>>>>>>>>>>>>> "ovirt_build_extra_flags
> >>>>>>>>>>>>>>>>>> -gs
> >>>>>>>>>>>>>>>>>> ${CI_MAVEN_SETTINGS}
> >>>>>>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera
> >>>>>>>>>>>>>>>>>> -Dgwt.logLevel=TRACE -X" --rebuild
> >>>>>>>>>>>>>>>>>> rpmbuild/SRPMS/*.src.rpm
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Let me know if you need anything else.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Vojtech
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>>>>>>>>> From: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>>>>>>>>>>>>>>> To: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>>>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>>>>>>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko"
> >>>>>>>>>>>>>>>>>>>> <knesenko at redhat.com>,
> >>>>>>>>>>>>>>>>>>>> "arch" <arch at ovirt.org>
> >>>>>>>>>>>>>>>>>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM
> >>>>>>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> In your log I see following error:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> ...
> >>>>>>>>>>>>>>>>>>>> [INFO] Computing all possible rebind results for
> >>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> >>>>>>>>>>>>>>>>>>>> [INFO] Rebinding
> >>>>>>>>>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
> >>>>>>>>>>>>>>>>>>>> [INFO] Invoking generator
> >>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
> >>>>>>>>>>>>>>>>>>>> [INFO] [ERROR] Generator
> >>>>>>>>>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
> >>>>>>>>>>>>>>>>>>>> threw
> >>>>>>>>>>>>>>>>>>>> an exception while rebinding
> >>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> >>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever
> >>>>>>>>>>>>>>>>>>>> computed
> >>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>> removeBrickCommitRequired
> >>>>>>>>>>>>>>>>>>>> [INFO] at
> >>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
> >>>>>>>>>>>>>>>>>>>> ...
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I checked again logs at
> >>>>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host/consoleFull
> >>>>>>>>>>>>>>>>>>>> and indeed there is a similar error:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> ...
> >>>>>>>>>>>>>>>>>>>> [INFO] Computing all possible rebind results for
> >>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> >>>>>>>>>>>>>>>>>>>> [INFO] Rebinding
> >>>>>>>>>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
> >>>>>>>>>>>>>>>>>>>> [INFO] Invoking generator
> >>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
> >>>>>>>>>>>>>>>>>>>> [INFO] [ERROR] Generator
> >>>>>>>>>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
> >>>>>>>>>>>>>>>>>>>> threw
> >>>>>>>>>>>>>>>>>>>> an exception while rebinding
> >>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> >>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException
> >>>>>>>>>>>>>>>>>>>> [INFO] at
> >>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
> >>>>>>>>>>>>>>>>>>>> ...
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> NullPointerException when processing
> >>>>>>>>>>>>>>>>>>>> ApplicationResources
> >>>>>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>> occuring
> >>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>> different places (ImageResourceGenerator vs.
> >>>>>>>>>>>>>>>>>>>> InlineResourceContext)
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Now I understand why Maven build fails without any
> >>>>>>>>>>>>>>>>>>>> details:
> >>>>>>>>>>>>>>>>>>>> - GWT deferred binding fails for ApplicationResources
> >>>>>>>>>>>>>>>>>>>> (i.e.
> >>>>>>>>>>>>>>>>>>>> uncaught
> >>>>>>>>>>>>>>>>>>>> NPE)
> >>>>>>>>>>>>>>>>>>>> - GWT compilation proceeds, fails to link with
> >>>>>>>>>>>>>>>>>>>> ApplicationResources
> >>>>>>>>>>>>>>>>>>>> - GWT compilation task fails with non-OK status code
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> So the issue here is GWT deferred binding generator
> >>>>>>>>>>>>>>>>>>>> (InlineClientBundleGenerator) failure during GWT
> >>>>>>>>>>>>>>>>>>>> compilation.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> GWT compiler writes extra Java files during deferred
> >>>>>>>>>>>>>>>>>>>> binding
> >>>>>>>>>>>>>>>>>>>> phase, so this could point to issue with local
> >>>>>>>>>>>>>>>>>>>> environment..
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I will try to investigate this and look at GWT source
> >>>>>>>>>>>>>>>>>>>> code.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>> Sandro Bonazzola
> >>>>>>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community
> >>>>>>>>>>>>>>>>>> collaboration.
> >>>>>>>>>>>>>>>>>> See how it works at redhat.com
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>> Sandro Bonazzola
> >>>>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community
> >>>>>>>>>>>>>>>> collaboration.
> >>>>>>>>>>>>>>>> See how it works at redhat.com
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> Sandro Bonazzola
> >>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community
> >>>>>>>>>>>>>> collaboration.
> >>>>>>>>>>>>>> See how it works at redhat.com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Sandro Bonazzola
> >>>>>>>>>>>> Better technology. Faster innovation. Powered by community
> >>>>>>>>>>>> collaboration.
> >>>>>>>>>>>> See how it works at redhat.com
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Sandro Bonazzola
> >>>>>>>>>> Better technology. Faster innovation. Powered by community
> >>>>>>>>>> collaboration.
> >>>>>>>>>> See how it works at redhat.com
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Sandro Bonazzola
> >>>>>>> Better technology. Faster innovation. Powered by community
> >>>>>>> collaboration.
> >>>>>>> See how it works at redhat.com
> >>>>>>>
> >>>>>> _______________________________________________
> >>>>>> Arch mailing list
> >>>>>> Arch at ovirt.org
> >>>>>> http://lists.ovirt.org/mailman/listinfo/arch
> >>>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Sandro Bonazzola
> >>>> Better technology. Faster innovation. Powered by community
> >>>> collaboration.
> >>>> See how it works at redhat.com
> >>>> _______________________________________________
> >>>> Arch mailing list
> >>>> Arch at ovirt.org
> >>>> http://lists.ovirt.org/mailman/listinfo/arch
> >>>>
> >>
> >>
> >> --
> >> Sandro Bonazzola
> >> Better technology. Faster innovation. Powered by community collaboration.
> >> See how it works at redhat.com
> >>
>
>
> --
> Sandro Bonazzola
> Better technology. Faster innovation. Powered by community collaboration.
> See how it works at redhat.com
>
More information about the Arch
mailing list