[urgent] Re: ovirt-engine build failure
Vojtech Szocs
vszocs at redhat.com
Thu Feb 27 13:55:08 UTC 2014
----- Original Message -----
> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> To: "Vojtech Szocs" <vszocs at redhat.com>
> Cc: "Tomas Jelinek" <tjelinek at redhat.com>, "Juan Hernandez" <jhernand at redhat.com>, "Kiril Nesenko"
> <knesenko at redhat.com>, "arch" <arch at ovirt.org>
> Sent: Wednesday, February 26, 2014 5:37:20 PM
> Subject: Re: [urgent] Re: ovirt-engine build failure
>
> Il 26/02/2014 16:40, Vojtech Szocs ha scritto:
> > (top posting)
> >
> > I did some preliminary tests on my local machine, invoking "mvn clean
> > install -Pgwt-admin" command to build (only) WebAdmin from source.
> >
> > Default permutation settings: all (5) browsers, one locale = 5 total
> > permutations to compile
> >
> > Tested two different strategies (machine with 16G RAM / 8G swap, quad-core
> > CPU):
> > A, localWorkers=X (default external-JVM worker factory)
> > -Dgwt.compiler.localWorkers=X
> > B, localWorkers=X + maxThreads=X (thread-based worker factory)
> > -Dgwt.compiler.localWorkers=X
> > -Dgwt.jjs.permutationWorkerFactory=com.google.gwt.dev.ThreadedPermutationWorkerFactory
> > -Dgwt.jjs.maxThreads=X
> >
> > A -> this is what we use today
> > B -> this is possible alternative
> >
> > X=4 [5 perms]
> > A, initial 2.6 GiB, peak 13.5 GiB
> > Total time: 5:25.851s
> > B, initial 2.6 GiB, peak 13.1 GiB
> > Total time: 5:11.238s
> >
> > X=1 [5 perms]
> > A, initial 2.6 GiB, peak 7.1 GiB
> > Total time: 7:00.601s
> > B, initial 2.6 GiB, peak 7.4 GiB
> > Total time: 7:07.458s
> >
> > Now switched to permutation settings: all (5) browsers, 3 locales = 15
> > total permutations to compile
> >
> > (Used swapoff/swapon to force swap clear after each test run.)
> >
> > X=4 [15 perms]
> > A, initial 2.6 GiB, peak physical max + swap ~4.3 GiB
> > Total time: 11:39.086s
> > B, initial 2.6 GiB, peak physical max + swap ~3.5 GiB
> > Total time: 10:59.966s
> >
> > X=1 [15 perms]
> > A, initial 2.6 GiB, peak 7.7 GiB
> > Total time: 17:39.103s
> >
> > B, initial 2.6 GiB. peak 7.9 GiB
> > Total time: 17:16.314s
> >
> > For X=1, both strategies (A & B) have very similar results.
> >
> > For X=4 (more than one worker), thread-based strategy seems slightly
> > better, but still we have huge RAM consumption + max opened files quite
> > high.
> >
> > I think the only real solution is to cut off (eliminate) locale permutation
> > vector to reduce amount of GWT permutations. It's on my task list.
> >
>
>
> Great!
> BTW, raising max opened files to 10240 worked, thanks.
Submitted patch for master: http://gerrit.ovirt.org/#/c/25163/
Details in commit message, short summary:
- GWT compiler will spawn thread(s) instead of Java sub-process(es) to compile specific permutation(s)
- max worker thread upper bound is set to 4
- you can use gwt.compiler.localWorkers to control max worker thread value,
i.e. -Dgwt.compiler.localWorkers=2 means max 2 worker threads -> min(2,4)
However, I don't expect too much out of this optimization; eliminating
GWT permutations is something we'll consider as long-term improvement.
Vojtech
>
>
> > Vojtech
> >
> >
> > ----- Original Message -----
> >> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >> To: "Vojtech Szocs" <vszocs at redhat.com>
> >> Cc: "Tomas Jelinek" <tjelinek at redhat.com>, "Juan Hernandez"
> >> <jhernand at redhat.com>, "Kiril Nesenko"
> >> <knesenko at redhat.com>, "arch" <arch at ovirt.org>
> >> Sent: Wednesday, February 26, 2014 11:58:29 AM
> >> Subject: Re: [urgent] Re: ovirt-engine build failure
> >>
> >> Il 26/02/2014 11:51, Vojtech Szocs ha scritto:
> >>>
> >>>
> >>> ----- Original Message -----
> >>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>> To: "Tomas Jelinek" <tjelinek at redhat.com>
> >>>> Cc: "Vojtech Szocs" <vszocs at redhat.com>, "Juan Hernandez"
> >>>> <jhernand at redhat.com>, "Kiril Nesenko"
> >>>> <knesenko at redhat.com>, "arch" <arch at ovirt.org>
> >>>> Sent: Wednesday, February 26, 2014 10:22:22 AM
> >>>> Subject: Re: [urgent] Re: ovirt-engine build failure
> >>>>
> >>>> Il 26/02/2014 09:06, Tomas Jelinek ha scritto:
> >>>>> Hey,
> >>>>>
> >>>>> from this logs it seems that the root cause is:
> >>>>> [ERROR] java.io.FileNotFoundException:
> >>>>> /home/jenkins/workspace/manual-build-tarball/label/centos6-host/rpmbuild/BUILD/ovirt-engine-3.5.0/frontend/webadmin/modules/webadmin/target/tmp/ImageResourceGenerator4201639982561998434.png
> >>>>> (Too many open files)
> >>>>>
> >>>>> e.g. the OS does not allow to open all the files the gwt compilation
> >>>>> needs.
> >>>>> Could you please try to increase the limit of max opened files on the
> >>>>> host
> >>>>> and try to rebuild?
> >>>>
> >>>> David is increasing max open files. We'll give it another run.
> >>>> I really think that having to reduce localWorkers to 1 for having decent
> >>>> build requirements on RAM and still hit too many opened files is really
> >>>> bad.
> >>>> Any chance of lowering build requirements in terms of RAM and opened
> >>>> files?
> >>>
> >>> In general, GWT compiler generates/reads lots of intermediate files
> >>> during
> >>> its runtime. IIRC, this file amount is proportional to number of
> >>> permutations
> >>> being compiled, i.e. num_of_browsers x num_of_locales. This is why we
> >>> don't
> >>> hit "Too many open files" in small-scale builds (i.e. development) but we
> >>> do
> >>> hit it on large-scale builds (i.e. all_browsers x all_locales nightly
> >>> build).
> >>>
> >>> But is increasing max open files for GWT compiler really that big of a
> >>> deal?
> >>> (i.e. we know that compiler needs to process lots of files per each
> >>> permutation)
> >>
> >> No, it's not that big of a deal *for our packaging* but since we've people
> >> on
> >> Gentoo building engine
> >> on their own systems, it may impact all of them...
> >>
> >>>
> >>> Alternatively, we could implement "remote GWT worker grid" -> offload
> >>> compilation
> >>> for each specific (browser x locale) permutation to a remote machine,
> >>> this
> >>> would
> >>> require some extra Java implementation though.
> >>>
> >>> I agree on excessive RAM usage, I'm currently experimenting with some
> >>> compiler
> >>> options, I'll share results when I have something.
> >>>
> >>> Another approach we could take (something I'll discuss with my peers) is
> >>> the
> >>> idea of eliminating locale vector entirely -> re-implement i18n support
> >>> in
> >>> UI
> >>> in a way that for X browsers, there will be only X total permutations
> >>> (i.e.
> >>> not X x num_of_locales). This would reduce RAM usage && compilation time
> >>> &&
> >>> perhaps max files opened too.
> >>
> >> +1 on whatever can reduce build time and build requirements.
> >>
> >>
> >>>
> >>>>
> >>>>
> >>>>>
> >>>>> ----- Original Message -----
> >>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>> Cc: "Juan Hernandez" <jhernand at redhat.com>, "Kiril Nesenko"
> >>>>>> <knesenko at redhat.com>, "arch" <arch at ovirt.org>
> >>>>>> Sent: Wednesday, February 26, 2014 8:37:07 AM
> >>>>>> Subject: Re: [urgent] Re: ovirt-engine build failure
> >>>>>>
> >>>>>> Il 25/02/2014 16:52, Vojtech Szocs ha scritto:
> >>>>>>>
> >>>>>>>
> >>>>>>> ----- Original Message -----
> >>>>>>>> From: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>>> To: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>>> Cc: "Juan Hernandez" <jhernand at redhat.com>, "Kiril Nesenko"
> >>>>>>>> <knesenko at redhat.com>, "arch" <arch at ovirt.org>
> >>>>>>>> Sent: Tuesday, February 25, 2014 1:05:35 PM
> >>>>>>>> Subject: Re: [urgent] Re: ovirt-engine build failure
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ----- Original Message -----
> >>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>>>> Cc: "Juan Hernandez" <jhernand at redhat.com>, "arch"
> >>>>>>>>> <arch at ovirt.org>,
> >>>>>>>>> "Kiril
> >>>>>>>>> Nesenko" <knesenko at redhat.com>
> >>>>>>>>> Sent: Tuesday, February 25, 2014 9:02:11 AM
> >>>>>>>>> Subject: Re: [urgent] Re: ovirt-engine build failure
> >>>>>>>>>
> >>>>>>>>> Il 25/02/2014 08:47, Sandro Bonazzola ha scritto:
> >>>>>>>>>> Il 24/02/2014 18:28, Vojtech Szocs ha scritto:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> >>>>>>>>>>>> "arch" <arch at ovirt.org>
> >>>>>>>>>>>> Sent: Monday, February 24, 2014 9:08:09 AM
> >>>>>>>>>>>> Subject: [urgent] Re: ovirt-engine build failure
> >>>>>>>>>>>>
> >>>>>>>>>>>> Top posting,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Please give this task really urgent priority, as you can see:
> >>>>>>>>>>>>
> >>>>>>>>>>>> http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/
> >>>>>>>>>>>> and
> >>>>>>>>>>>> http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/
> >>>>>>>>>>>>
> >>>>>>>>>>>> are failing consistently so we don't have nightly builds.
> >>>>>>>>>>>
> >>>>>>>>>>> (CC'ing Einav)
> >>>>>>>>>>>
> >>>>>>>>>>> Attached TEST patch, please apply & rebuild ovirt-engine as
> >>>>>>>>>>> usual.
> >>>>>>>>>>>
> >>>>>>>>>>> This patch should cause following output during GWT compilation:
> >>>>>>>>>>>
> >>>>>>>>>>> [ERROR] ######### Util.readURLAsBytes
> >>>>>>>>>>> file:/home/vszocs/work/ovirt-engine/frontend/webadmin/modules/webadmin/target/generated-sources/gwt/org/ovirt/engine/ui/webadmin/images/search_button.png
> >>>>>>>>>>>
> >>>>>>>>>>> If IOException occurs, exception stacktrace will be printed right
> >>>>>>>>>>> below
> >>>>>>>>>>> that line.
> >>>>>>>>>>>
> >>>>>>>>>>> Note: please ignore the [ERROR] prefix, it's a technical issue,
> >>>>>>>>>>> more
> >>>>>>>>>>> details at
> >>>>>>>>>>> https://www.mail-archive.com/engine-devel@ovirt.org/msg05735.html
> >>>>>>>>>>>
> >>>>>>>>>>> I just did a local upstream build with TEST patch applied, GWT
> >>>>>>>>>>> compilation
> >>>>>>>>>>> passed OK.
> >>>>>>>>>>
> >>>>>>>>>> build job running with your patch here:
> >>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/235/
> >>>>>>>>>> waiting for a free executor right now.
> >>>>>>>>>> Let's see how it goes.
> >>>>>>>>>
> >>>>>>>>> Failed on CentOS:
> >>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/235/label=centos6-host/consoleFull
> >>>>>>>>
> >>>>>>>> Hm, in that ^^ build, following exceptions occured:
> >>>>>>>>
> >>>>>>>> 1.
> >>>>>>>> [INFO] java.lang.NullPointerException
> >>>>>>>> [INFO] at
> >>>>>>>> com.google.gwt.dev.util.Util.computeStrongName(Util.java:172)
> >>>>>>>>
> >>>>>>>> 2.
> >>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed
> >>>>>>>> for
> >>>>>>>> downDisabledImage
> >>>>>>>> [INFO] at
> >>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475
> >>>>>>>>
> >>>>>>>> 1. is caused by Util#readURLAsBytes but I don't see any IOException
> >>>>>>>> in
> >>>>>>>> the
> >>>>>>>> logs
> >>>>>>>> 2. is just follow-up consequence of 1. (i.e. image metadata missing
> >>>>>>>> in
> >>>>>>>> GWT
> >>>>>>>> image cache)
> >>>>>>>>
> >>>>>>>> Now I noticed I missed one possible IOException inside
> >>>>>>>> Util#readURLConnectionAsBytes ...
> >>>>>>>>
> >>>>>>>> Long story short, I'll send updated TEST patch soon, apologies for
> >>>>>>>> the
> >>>>>>>> inconvenience.
> >>>>>>>
> >>>>>>> TEST-v2 patch attached. Please apply & rebuild ovirt-engine as usual.
> >>>>>>
> >>>>>> Done, here is the console output:
> >>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/label=centos6-host/239/consoleFull
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Il 20/02/2014 13:34, Vojtech Szocs ha scritto:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> >>>>>>>>>>>>>> "arch" <arch at ovirt.org>
> >>>>>>>>>>>>>> Sent: Thursday, February 20, 2014 10:02:00 AM
> >>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Il 19/02/2014 16:51, Vojtech Szocs ha scritto:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko"
> >>>>>>>>>>>>>>>> <knesenko at redhat.com>,
> >>>>>>>>>>>>>>>> "arch" <arch at ovirt.org>
> >>>>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 2:43:44 PM
> >>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Il 19/02/2014 14:41, Vojtech Szocs ha scritto:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>>>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>>>>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko"
> >>>>>>>>>>>>>>>>>> <knesenko at redhat.com>,
> >>>>>>>>>>>>>>>>>> "arch" <arch at ovirt.org>
> >>>>>>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 2:23:16 PM
> >>>>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Il 19/02/2014 14:19, Vojtech Szocs ha scritto:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>>>>>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>>>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>>>>>>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko"
> >>>>>>>>>>>>>>>>>>>> <knesenko at redhat.com>,
> >>>>>>>>>>>>>>>>>>>> "arch" <arch at ovirt.org>
> >>>>>>>>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 1:18:28 PM
> >>>>>>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto:
> >>>>>>>>>>>>>>>>>>>>> Hi, I checked GWT sources:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect
> >>>>>>>>>>>>>>>>>>>>>> ever
> >>>>>>>>>>>>>>>>>>>>>> computed
> >>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>> removeBrickCommitRequired
> >>>>>>>>>>>>>>>>>>>>>> [INFO] at
> >>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> This happened because internal image cache for bundled
> >>>>>>>>>>>>>>>>>>>>> (composite
> >>>>>>>>>>>>>>>>>>>>> PNG)
> >>>>>>>>>>>>>>>>>>>>> images was corrput.
> >>>>>>>>>>>>>>>>>>>>> It seems to be caused by disk I/O error while reading
> >>>>>>>>>>>>>>>>>>>>> file
> >>>>>>>>>>>>>>>>>>>>> images/glusterAsyncTasks/remove_brick_commit_required.png
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException
> >>>>>>>>>>>>>>>>>>>>>> [INFO] at
> >>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> This happened because image data was null, again
> >>>>>>>>>>>>>>>>>>>>> possible
> >>>>>>>>>>>>>>>>>>>>> disk
> >>>>>>>>>>>>>>>>>>>>> I/O
> >>>>>>>>>>>>>>>>>>>>> error
> >>>>>>>>>>>>>>>>>>>>> while reading image file.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Can we enable "TRACE" logging during GWT compilation?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> -Dgwt.logLevel=TRACE
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Building master right now with your suggestions here:
> >>>>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Looking at
> >>>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host/console
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> So far I see no errors.. There should be total 5 browsers
> >>>>>>>>>>>>>>>>>>> x
> >>>>>>>>>>>>>>>>>>> 8
> >>>>>>>>>>>>>>>>>>> locales
> >>>>>>>>>>>>>>>>>>> =
> >>>>>>>>>>>>>>>>>>> 40
> >>>>>>>>>>>>>>>>>>> GWT permutations.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> So no changes with and without
> >>>>>>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera.
> >>>>>>>>>>>>>>>>>> We've still 40 permutations. Maybe you defined
> >>>>>>>>>>>>>>>>>> userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the
> >>>>>>>>>>>>>>>>>> POM
> >>>>>>>>>>>>>>>>>> files?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> You are right. I checked frontend/webadmin/modules/pom.xml
> >>>>>>>>>>>>>>>>> ->
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> "gwt.userAgent"
> >>>>>>>>>>>>>>>>> property is copied into "locale" GWT property in
> >>>>>>>>>>>>>>>>> {WebAdmin,UserPortal}.gwt.xml
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Ok, so I'll remove
> >>>>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera
> >>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>> build
> >>>>>>>>>>>>>>>> command.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Agreed.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> This:
> >>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/console
> >>>>>>>>>>>>>>>>>> failed, you can look at it, it has -Dgwt.logLevel=TRACE
> >>>>>>>>>>>>>>>>>> -X.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hm, it contains:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> [DEBUG] (f) logLevel = INFO
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Somehow the TRACE log level for GWT wasn't applied.. not
> >>>>>>>>>>>>>>>>> sure
> >>>>>>>>>>>>>>>>> why.
> >>>>>>>>>>>>>>>>> Let
> >>>>>>>>>>>>>>>>> me
> >>>>>>>>>>>>>>>>> check
> >>>>>>>>>>>>>>>>> this on local build.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Ok, let me know if I've to change command line.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> When it's merged, setting "gwt.logLevel" will work (as you
> >>>>>>>>>>>>>>> did
> >>>>>>>>>>>>>>> before)
> >>>>>>>>>>>>>>> and we'll be able to see more details in the build log.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Here is the new build with your patch merged:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/consoleFull
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks. I did some more investigation on this.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Again it's NullPointerException when rebinding
> >>>>>>>>>>>>> ApplicationResources,
> >>>>>>>>>>>>> but
> >>>>>>>>>>>>> this time for different image file:
> >>>>>>>>>>>>> ApplicationResources.wrenchImage
> >>>>>>>>>>>>> ("images/wrench.png").
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Looking at stacktrace & GWT sources, the problematic code seems
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>> be
> >>>>>>>>>>>>> (AbstractResourceContext:74):
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> byte[] bytes = Util.readURLAsBytes(resource);
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> where "resource" is (ImageResourceGenerator:311):
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> localized.getUrl(); // LocalizedImage localized
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Note: in oVirt UI we don't use localized images (i.e.
> >>>>>>>>>>>>> "wrench_fr_FR.png")
> >>>>>>>>>>>>> so "localized" in this case points to "wrench.png".
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> So the problem seems to be reading this file:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Since we're having many GWT permutations, this file ^^ (along
> >>>>>>>>>>>>> with
> >>>>>>>>>>>>> similar
> >>>>>>>>>>>>> ones) is attempted to be read multiple times *from the same
> >>>>>>>>>>>>> (single)
> >>>>>>>>>>>>> GWT
> >>>>>>>>>>>>> compiler Java process, before actual permutation worker Java
> >>>>>>>>>>>>> sub-processes
> >>>>>>>>>>>>> are spawned*. (More precisely, above NullPointerException
> >>>>>>>>>>>>> happens
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>> deferred binding phase which precedes actual GWT compilation
> >>>>>>>>>>>>> per
> >>>>>>>>>>>>> each
> >>>>>>>>>>>>> permutation.)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I think the root cause is Util.readURLAsBytes method used
> >>>>>>>>>>>>> internally
> >>>>>>>>>>>>> by
> >>>>>>>>>>>>> GWT
> >>>>>>>>>>>>> compiler:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> public static byte[] readURLAsBytes(URL url) {
> >>>>>>>>>>>>> try {
> >>>>>>>>>>>>> URLConnection conn = url.openConnection();
> >>>>>>>>>>>>> conn.setUseCaches(false);
> >>>>>>>>>>>>> return readURLConnectionAsBytes(conn);
> >>>>>>>>>>>>> } catch (IOException e) {
> >>>>>>>>>>>>> return null; // This causes NullPointerException, but
> >>>>>>>>>>>>> this
> >>>>>>>>>>>>> exception
> >>>>>>>>>>>>> isn't logged (wtf, GWT!)
> >>>>>>>>>>>>> }
> >>>>>>>>>>>>> }
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> To summarize, I'll need to write another patch which replaces
> >>>>>>>>>>>>> Util.readURLAsBytes implementation via AOP and prints
> >>>>>>>>>>>>> IOException
> >>>>>>>>>>>>> details.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> #
> >>>>>>>>>>>>>>>>>>>> CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml
> >>>>>>>>>>>>>>>>>>>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M
> >>>>>>>>>>>>>>>>>>>> -XX:MaxPermSize=1024m
> >>>>>>>>>>>>>>>>>>>> -Dgwt.compiler.localWorkers=1"
> >>>>>>>>>>>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}"
> >>>>>>>>>>>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D
> >>>>>>>>>>>>>>>>>>>> "ovirt_build_extra_flags
> >>>>>>>>>>>>>>>>>>>> -gs
> >>>>>>>>>>>>>>>>>>>> ${CI_MAVEN_SETTINGS}
> >>>>>>>>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera
> >>>>>>>>>>>>>>>>>>>> -Dgwt.logLevel=TRACE -X" --rebuild
> >>>>>>>>>>>>>>>>>>>> rpmbuild/SRPMS/*.src.rpm
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Let me know if you need anything else.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Vojtech
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>>>>>>>>>>> From: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>>>>>>>>>>>>>>>>> To: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>>>>>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>>>>>>>>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko"
> >>>>>>>>>>>>>>>>>>>>>> <knesenko at redhat.com>,
> >>>>>>>>>>>>>>>>>>>>>> "arch" <arch at ovirt.org>
> >>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM
> >>>>>>>>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> In your log I see following error:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> ...
> >>>>>>>>>>>>>>>>>>>>>> [INFO] Computing all possible rebind results for
> >>>>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> >>>>>>>>>>>>>>>>>>>>>> [INFO] Rebinding
> >>>>>>>>>>>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
> >>>>>>>>>>>>>>>>>>>>>> [INFO] Invoking generator
> >>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
> >>>>>>>>>>>>>>>>>>>>>> [INFO] [ERROR] Generator
> >>>>>>>>>>>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
> >>>>>>>>>>>>>>>>>>>>>> threw
> >>>>>>>>>>>>>>>>>>>>>> an exception while rebinding
> >>>>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> >>>>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect
> >>>>>>>>>>>>>>>>>>>>>> ever
> >>>>>>>>>>>>>>>>>>>>>> computed
> >>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>> removeBrickCommitRequired
> >>>>>>>>>>>>>>>>>>>>>> [INFO] at
> >>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
> >>>>>>>>>>>>>>>>>>>>>> ...
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I checked again logs at
> >>>>>>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host/consoleFull
> >>>>>>>>>>>>>>>>>>>>>> and indeed there is a similar error:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> ...
> >>>>>>>>>>>>>>>>>>>>>> [INFO] Computing all possible rebind results for
> >>>>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> >>>>>>>>>>>>>>>>>>>>>> [INFO] Rebinding
> >>>>>>>>>>>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
> >>>>>>>>>>>>>>>>>>>>>> [INFO] Invoking generator
> >>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
> >>>>>>>>>>>>>>>>>>>>>> [INFO] [ERROR] Generator
> >>>>>>>>>>>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
> >>>>>>>>>>>>>>>>>>>>>> threw
> >>>>>>>>>>>>>>>>>>>>>> an exception while rebinding
> >>>>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> >>>>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException
> >>>>>>>>>>>>>>>>>>>>>> [INFO] at
> >>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
> >>>>>>>>>>>>>>>>>>>>>> ...
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> NullPointerException when processing
> >>>>>>>>>>>>>>>>>>>>>> ApplicationResources
> >>>>>>>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>>> occuring
> >>>>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>> different places (ImageResourceGenerator vs.
> >>>>>>>>>>>>>>>>>>>>>> InlineResourceContext)
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Now I understand why Maven build fails without any
> >>>>>>>>>>>>>>>>>>>>>> details:
> >>>>>>>>>>>>>>>>>>>>>> - GWT deferred binding fails for ApplicationResources
> >>>>>>>>>>>>>>>>>>>>>> (i.e.
> >>>>>>>>>>>>>>>>>>>>>> uncaught
> >>>>>>>>>>>>>>>>>>>>>> NPE)
> >>>>>>>>>>>>>>>>>>>>>> - GWT compilation proceeds, fails to link with
> >>>>>>>>>>>>>>>>>>>>>> ApplicationResources
> >>>>>>>>>>>>>>>>>>>>>> - GWT compilation task fails with non-OK status code
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> So the issue here is GWT deferred binding generator
> >>>>>>>>>>>>>>>>>>>>>> (InlineClientBundleGenerator) failure during GWT
> >>>>>>>>>>>>>>>>>>>>>> compilation.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> GWT compiler writes extra Java files during deferred
> >>>>>>>>>>>>>>>>>>>>>> binding
> >>>>>>>>>>>>>>>>>>>>>> phase, so this could point to issue with local
> >>>>>>>>>>>>>>>>>>>>>> environment..
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I will try to investigate this and look at GWT source
> >>>>>>>>>>>>>>>>>>>>>> code.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>> Sandro Bonazzola
> >>>>>>>>>>>>>>>>>>>> Better technology. Faster innovation. Powered by
> >>>>>>>>>>>>>>>>>>>> community
> >>>>>>>>>>>>>>>>>>>> collaboration.
> >>>>>>>>>>>>>>>>>>>> See how it works at redhat.com
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>> Sandro Bonazzola
> >>>>>>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community
> >>>>>>>>>>>>>>>>>> collaboration.
> >>>>>>>>>>>>>>>>>> See how it works at redhat.com
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>> Sandro Bonazzola
> >>>>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community
> >>>>>>>>>>>>>>>> collaboration.
> >>>>>>>>>>>>>>>> See how it works at redhat.com
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> Sandro Bonazzola
> >>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community
> >>>>>>>>>>>>>> collaboration.
> >>>>>>>>>>>>>> See how it works at redhat.com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Sandro Bonazzola
> >>>>>>>>>>>> Better technology. Faster innovation. Powered by community
> >>>>>>>>>>>> collaboration.
> >>>>>>>>>>>> See how it works at redhat.com
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Sandro Bonazzola
> >>>>>>>>> Better technology. Faster innovation. Powered by community
> >>>>>>>>> collaboration.
> >>>>>>>>> See how it works at redhat.com
> >>>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> Arch mailing list
> >>>>>>>> Arch at ovirt.org
> >>>>>>>> http://lists.ovirt.org/mailman/listinfo/arch
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Sandro Bonazzola
> >>>>>> Better technology. Faster innovation. Powered by community
> >>>>>> collaboration.
> >>>>>> See how it works at redhat.com
> >>>>>> _______________________________________________
> >>>>>> Arch mailing list
> >>>>>> Arch at ovirt.org
> >>>>>> http://lists.ovirt.org/mailman/listinfo/arch
> >>>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Sandro Bonazzola
> >>>> Better technology. Faster innovation. Powered by community
> >>>> collaboration.
> >>>> See how it works at redhat.com
> >>>>
> >>
> >>
> >> --
> >> Sandro Bonazzola
> >> Better technology. Faster innovation. Powered by community collaboration.
> >> See how it works at redhat.com
> >>
>
>
> --
> Sandro Bonazzola
> Better technology. Faster innovation. Powered by community collaboration.
> See how it works at redhat.com
>
More information about the Arch
mailing list