[urgent] Re: ovirt-engine build failure
Sandro Bonazzola
sbonazzo at redhat.com
Wed Feb 26 16:37:20 UTC 2014
Il 26/02/2014 16:40, Vojtech Szocs ha scritto:
> (top posting)
>
> I did some preliminary tests on my local machine, invoking "mvn clean install -Pgwt-admin" command to build (only) WebAdmin from source.
>
> Default permutation settings: all (5) browsers, one locale = 5 total permutations to compile
>
> Tested two different strategies (machine with 16G RAM / 8G swap, quad-core CPU):
> A, localWorkers=X (default external-JVM worker factory)
> -Dgwt.compiler.localWorkers=X
> B, localWorkers=X + maxThreads=X (thread-based worker factory)
> -Dgwt.compiler.localWorkers=X -Dgwt.jjs.permutationWorkerFactory=com.google.gwt.dev.ThreadedPermutationWorkerFactory -Dgwt.jjs.maxThreads=X
>
> A -> this is what we use today
> B -> this is possible alternative
>
> X=4 [5 perms]
> A, initial 2.6 GiB, peak 13.5 GiB
> Total time: 5:25.851s
> B, initial 2.6 GiB, peak 13.1 GiB
> Total time: 5:11.238s
>
> X=1 [5 perms]
> A, initial 2.6 GiB, peak 7.1 GiB
> Total time: 7:00.601s
> B, initial 2.6 GiB, peak 7.4 GiB
> Total time: 7:07.458s
>
> Now switched to permutation settings: all (5) browsers, 3 locales = 15 total permutations to compile
>
> (Used swapoff/swapon to force swap clear after each test run.)
>
> X=4 [15 perms]
> A, initial 2.6 GiB, peak physical max + swap ~4.3 GiB
> Total time: 11:39.086s
> B, initial 2.6 GiB, peak physical max + swap ~3.5 GiB
> Total time: 10:59.966s
>
> X=1 [15 perms]
> A, initial 2.6 GiB, peak 7.7 GiB
> Total time: 17:39.103s
>
> B, initial 2.6 GiB. peak 7.9 GiB
> Total time: 17:16.314s
>
> For X=1, both strategies (A & B) have very similar results.
>
> For X=4 (more than one worker), thread-based strategy seems slightly better, but still we have huge RAM consumption + max opened files quite high.
>
> I think the only real solution is to cut off (eliminate) locale permutation vector to reduce amount of GWT permutations. It's on my task list.
>
Great!
BTW, raising max opened files to 10240 worked, thanks.
> Vojtech
>
>
> ----- Original Message -----
>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>> To: "Vojtech Szocs" <vszocs at redhat.com>
>> Cc: "Tomas Jelinek" <tjelinek at redhat.com>, "Juan Hernandez" <jhernand at redhat.com>, "Kiril Nesenko"
>> <knesenko at redhat.com>, "arch" <arch at ovirt.org>
>> Sent: Wednesday, February 26, 2014 11:58:29 AM
>> Subject: Re: [urgent] Re: ovirt-engine build failure
>>
>> Il 26/02/2014 11:51, Vojtech Szocs ha scritto:
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>> To: "Tomas Jelinek" <tjelinek at redhat.com>
>>>> Cc: "Vojtech Szocs" <vszocs at redhat.com>, "Juan Hernandez"
>>>> <jhernand at redhat.com>, "Kiril Nesenko"
>>>> <knesenko at redhat.com>, "arch" <arch at ovirt.org>
>>>> Sent: Wednesday, February 26, 2014 10:22:22 AM
>>>> Subject: Re: [urgent] Re: ovirt-engine build failure
>>>>
>>>> Il 26/02/2014 09:06, Tomas Jelinek ha scritto:
>>>>> Hey,
>>>>>
>>>>> from this logs it seems that the root cause is:
>>>>> [ERROR] java.io.FileNotFoundException:
>>>>> /home/jenkins/workspace/manual-build-tarball/label/centos6-host/rpmbuild/BUILD/ovirt-engine-3.5.0/frontend/webadmin/modules/webadmin/target/tmp/ImageResourceGenerator4201639982561998434.png
>>>>> (Too many open files)
>>>>>
>>>>> e.g. the OS does not allow to open all the files the gwt compilation
>>>>> needs.
>>>>> Could you please try to increase the limit of max opened files on the
>>>>> host
>>>>> and try to rebuild?
>>>>
>>>> David is increasing max open files. We'll give it another run.
>>>> I really think that having to reduce localWorkers to 1 for having decent
>>>> build requirements on RAM and still hit too many opened files is really
>>>> bad.
>>>> Any chance of lowering build requirements in terms of RAM and opened
>>>> files?
>>>
>>> In general, GWT compiler generates/reads lots of intermediate files during
>>> its runtime. IIRC, this file amount is proportional to number of
>>> permutations
>>> being compiled, i.e. num_of_browsers x num_of_locales. This is why we don't
>>> hit "Too many open files" in small-scale builds (i.e. development) but we
>>> do
>>> hit it on large-scale builds (i.e. all_browsers x all_locales nightly
>>> build).
>>>
>>> But is increasing max open files for GWT compiler really that big of a
>>> deal?
>>> (i.e. we know that compiler needs to process lots of files per each
>>> permutation)
>>
>> No, it's not that big of a deal *for our packaging* but since we've people on
>> Gentoo building engine
>> on their own systems, it may impact all of them...
>>
>>>
>>> Alternatively, we could implement "remote GWT worker grid" -> offload
>>> compilation
>>> for each specific (browser x locale) permutation to a remote machine, this
>>> would
>>> require some extra Java implementation though.
>>>
>>> I agree on excessive RAM usage, I'm currently experimenting with some
>>> compiler
>>> options, I'll share results when I have something.
>>>
>>> Another approach we could take (something I'll discuss with my peers) is
>>> the
>>> idea of eliminating locale vector entirely -> re-implement i18n support in
>>> UI
>>> in a way that for X browsers, there will be only X total permutations (i.e.
>>> not X x num_of_locales). This would reduce RAM usage && compilation time &&
>>> perhaps max files opened too.
>>
>> +1 on whatever can reduce build time and build requirements.
>>
>>
>>>
>>>>
>>>>
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>> Cc: "Juan Hernandez" <jhernand at redhat.com>, "Kiril Nesenko"
>>>>>> <knesenko at redhat.com>, "arch" <arch at ovirt.org>
>>>>>> Sent: Wednesday, February 26, 2014 8:37:07 AM
>>>>>> Subject: Re: [urgent] Re: ovirt-engine build failure
>>>>>>
>>>>>> Il 25/02/2014 16:52, Vojtech Szocs ha scritto:
>>>>>>>
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>>> From: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>> To: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>> Cc: "Juan Hernandez" <jhernand at redhat.com>, "Kiril Nesenko"
>>>>>>>> <knesenko at redhat.com>, "arch" <arch at ovirt.org>
>>>>>>>> Sent: Tuesday, February 25, 2014 1:05:35 PM
>>>>>>>> Subject: Re: [urgent] Re: ovirt-engine build failure
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>>> Cc: "Juan Hernandez" <jhernand at redhat.com>, "arch" <arch at ovirt.org>,
>>>>>>>>> "Kiril
>>>>>>>>> Nesenko" <knesenko at redhat.com>
>>>>>>>>> Sent: Tuesday, February 25, 2014 9:02:11 AM
>>>>>>>>> Subject: Re: [urgent] Re: ovirt-engine build failure
>>>>>>>>>
>>>>>>>>> Il 25/02/2014 08:47, Sandro Bonazzola ha scritto:
>>>>>>>>>> Il 24/02/2014 18:28, Vojtech Szocs ha scritto:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>>>>>>>>> "arch" <arch at ovirt.org>
>>>>>>>>>>>> Sent: Monday, February 24, 2014 9:08:09 AM
>>>>>>>>>>>> Subject: [urgent] Re: ovirt-engine build failure
>>>>>>>>>>>>
>>>>>>>>>>>> Top posting,
>>>>>>>>>>>>
>>>>>>>>>>>> Please give this task really urgent priority, as you can see:
>>>>>>>>>>>>
>>>>>>>>>>>> http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/
>>>>>>>>>>>> and
>>>>>>>>>>>> http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/
>>>>>>>>>>>>
>>>>>>>>>>>> are failing consistently so we don't have nightly builds.
>>>>>>>>>>>
>>>>>>>>>>> (CC'ing Einav)
>>>>>>>>>>>
>>>>>>>>>>> Attached TEST patch, please apply & rebuild ovirt-engine as usual.
>>>>>>>>>>>
>>>>>>>>>>> This patch should cause following output during GWT compilation:
>>>>>>>>>>>
>>>>>>>>>>> [ERROR] ######### Util.readURLAsBytes
>>>>>>>>>>> file:/home/vszocs/work/ovirt-engine/frontend/webadmin/modules/webadmin/target/generated-sources/gwt/org/ovirt/engine/ui/webadmin/images/search_button.png
>>>>>>>>>>>
>>>>>>>>>>> If IOException occurs, exception stacktrace will be printed right
>>>>>>>>>>> below
>>>>>>>>>>> that line.
>>>>>>>>>>>
>>>>>>>>>>> Note: please ignore the [ERROR] prefix, it's a technical issue,
>>>>>>>>>>> more
>>>>>>>>>>> details at
>>>>>>>>>>> https://www.mail-archive.com/engine-devel@ovirt.org/msg05735.html
>>>>>>>>>>>
>>>>>>>>>>> I just did a local upstream build with TEST patch applied, GWT
>>>>>>>>>>> compilation
>>>>>>>>>>> passed OK.
>>>>>>>>>>
>>>>>>>>>> build job running with your patch here:
>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/235/
>>>>>>>>>> waiting for a free executor right now.
>>>>>>>>>> Let's see how it goes.
>>>>>>>>>
>>>>>>>>> Failed on CentOS:
>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/235/label=centos6-host/consoleFull
>>>>>>>>
>>>>>>>> Hm, in that ^^ build, following exceptions occured:
>>>>>>>>
>>>>>>>> 1.
>>>>>>>> [INFO] java.lang.NullPointerException
>>>>>>>> [INFO] at
>>>>>>>> com.google.gwt.dev.util.Util.computeStrongName(Util.java:172)
>>>>>>>>
>>>>>>>> 2.
>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed for
>>>>>>>> downDisabledImage
>>>>>>>> [INFO] at
>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475
>>>>>>>>
>>>>>>>> 1. is caused by Util#readURLAsBytes but I don't see any IOException in
>>>>>>>> the
>>>>>>>> logs
>>>>>>>> 2. is just follow-up consequence of 1. (i.e. image metadata missing in
>>>>>>>> GWT
>>>>>>>> image cache)
>>>>>>>>
>>>>>>>> Now I noticed I missed one possible IOException inside
>>>>>>>> Util#readURLConnectionAsBytes ...
>>>>>>>>
>>>>>>>> Long story short, I'll send updated TEST patch soon, apologies for the
>>>>>>>> inconvenience.
>>>>>>>
>>>>>>> TEST-v2 patch attached. Please apply & rebuild ovirt-engine as usual.
>>>>>>
>>>>>> Done, here is the console output:
>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/label=centos6-host/239/consoleFull
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Il 20/02/2014 13:34, Vojtech Szocs ha scritto:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>>>>>>>>>>> "arch" <arch at ovirt.org>
>>>>>>>>>>>>>> Sent: Thursday, February 20, 2014 10:02:00 AM
>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Il 19/02/2014 16:51, Vojtech Szocs ha scritto:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>>>>>>>>>>>>> "arch" <arch at ovirt.org>
>>>>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 2:43:44 PM
>>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Il 19/02/2014 14:41, Vojtech Szocs ha scritto:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko"
>>>>>>>>>>>>>>>>>> <knesenko at redhat.com>,
>>>>>>>>>>>>>>>>>> "arch" <arch at ovirt.org>
>>>>>>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 2:23:16 PM
>>>>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Il 19/02/2014 14:19, Vojtech Szocs ha scritto:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>>>>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>>>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko"
>>>>>>>>>>>>>>>>>>>> <knesenko at redhat.com>,
>>>>>>>>>>>>>>>>>>>> "arch" <arch at ovirt.org>
>>>>>>>>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 1:18:28 PM
>>>>>>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto:
>>>>>>>>>>>>>>>>>>>>> Hi, I checked GWT sources:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever
>>>>>>>>>>>>>>>>>>>>>> computed
>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>> removeBrickCommitRequired
>>>>>>>>>>>>>>>>>>>>>> [INFO] at
>>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> This happened because internal image cache for bundled
>>>>>>>>>>>>>>>>>>>>> (composite
>>>>>>>>>>>>>>>>>>>>> PNG)
>>>>>>>>>>>>>>>>>>>>> images was corrput.
>>>>>>>>>>>>>>>>>>>>> It seems to be caused by disk I/O error while reading
>>>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>>> images/glusterAsyncTasks/remove_brick_commit_required.png
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException
>>>>>>>>>>>>>>>>>>>>>> [INFO] at
>>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> This happened because image data was null, again possible
>>>>>>>>>>>>>>>>>>>>> disk
>>>>>>>>>>>>>>>>>>>>> I/O
>>>>>>>>>>>>>>>>>>>>> error
>>>>>>>>>>>>>>>>>>>>> while reading image file.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Can we enable "TRACE" logging during GWT compilation?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -Dgwt.logLevel=TRACE
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Building master right now with your suggestions here:
>>>>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Looking at
>>>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host/console
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> So far I see no errors.. There should be total 5 browsers x
>>>>>>>>>>>>>>>>>>> 8
>>>>>>>>>>>>>>>>>>> locales
>>>>>>>>>>>>>>>>>>> =
>>>>>>>>>>>>>>>>>>> 40
>>>>>>>>>>>>>>>>>>> GWT permutations.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> So no changes with and without
>>>>>>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera.
>>>>>>>>>>>>>>>>>> We've still 40 permutations. Maybe you defined
>>>>>>>>>>>>>>>>>> userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the POM
>>>>>>>>>>>>>>>>>> files?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> You are right. I checked frontend/webadmin/modules/pom.xml ->
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> "gwt.userAgent"
>>>>>>>>>>>>>>>>> property is copied into "locale" GWT property in
>>>>>>>>>>>>>>>>> {WebAdmin,UserPortal}.gwt.xml
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ok, so I'll remove
>>>>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera
>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>> build
>>>>>>>>>>>>>>>> command.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Agreed.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This:
>>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/console
>>>>>>>>>>>>>>>>>> failed, you can look at it, it has -Dgwt.logLevel=TRACE -X.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hm, it contains:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [DEBUG] (f) logLevel = INFO
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Somehow the TRACE log level for GWT wasn't applied.. not sure
>>>>>>>>>>>>>>>>> why.
>>>>>>>>>>>>>>>>> Let
>>>>>>>>>>>>>>>>> me
>>>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>> this on local build.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ok, let me know if I've to change command line.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When it's merged, setting "gwt.logLevel" will work (as you did
>>>>>>>>>>>>>>> before)
>>>>>>>>>>>>>>> and we'll be able to see more details in the build log.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here is the new build with your patch merged:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/consoleFull
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks. I did some more investigation on this.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Again it's NullPointerException when rebinding
>>>>>>>>>>>>> ApplicationResources,
>>>>>>>>>>>>> but
>>>>>>>>>>>>> this time for different image file:
>>>>>>>>>>>>> ApplicationResources.wrenchImage
>>>>>>>>>>>>> ("images/wrench.png").
>>>>>>>>>>>>>
>>>>>>>>>>>>> Looking at stacktrace & GWT sources, the problematic code seems
>>>>>>>>>>>>> to
>>>>>>>>>>>>> be
>>>>>>>>>>>>> (AbstractResourceContext:74):
>>>>>>>>>>>>>
>>>>>>>>>>>>> byte[] bytes = Util.readURLAsBytes(resource);
>>>>>>>>>>>>>
>>>>>>>>>>>>> where "resource" is (ImageResourceGenerator:311):
>>>>>>>>>>>>>
>>>>>>>>>>>>> localized.getUrl(); // LocalizedImage localized
>>>>>>>>>>>>>
>>>>>>>>>>>>> Note: in oVirt UI we don't use localized images (i.e.
>>>>>>>>>>>>> "wrench_fr_FR.png")
>>>>>>>>>>>>> so "localized" in this case points to "wrench.png".
>>>>>>>>>>>>>
>>>>>>>>>>>>> So the problem seems to be reading this file:
>>>>>>>>>>>>>
>>>>>>>>>>>>> frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png
>>>>>>>>>>>>>
>>>>>>>>>>>>> Since we're having many GWT permutations, this file ^^ (along
>>>>>>>>>>>>> with
>>>>>>>>>>>>> similar
>>>>>>>>>>>>> ones) is attempted to be read multiple times *from the same
>>>>>>>>>>>>> (single)
>>>>>>>>>>>>> GWT
>>>>>>>>>>>>> compiler Java process, before actual permutation worker Java
>>>>>>>>>>>>> sub-processes
>>>>>>>>>>>>> are spawned*. (More precisely, above NullPointerException happens
>>>>>>>>>>>>> at
>>>>>>>>>>>>> deferred binding phase which precedes actual GWT compilation per
>>>>>>>>>>>>> each
>>>>>>>>>>>>> permutation.)
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think the root cause is Util.readURLAsBytes method used
>>>>>>>>>>>>> internally
>>>>>>>>>>>>> by
>>>>>>>>>>>>> GWT
>>>>>>>>>>>>> compiler:
>>>>>>>>>>>>>
>>>>>>>>>>>>> public static byte[] readURLAsBytes(URL url) {
>>>>>>>>>>>>> try {
>>>>>>>>>>>>> URLConnection conn = url.openConnection();
>>>>>>>>>>>>> conn.setUseCaches(false);
>>>>>>>>>>>>> return readURLConnectionAsBytes(conn);
>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>> return null; // This causes NullPointerException, but this
>>>>>>>>>>>>> exception
>>>>>>>>>>>>> isn't logged (wtf, GWT!)
>>>>>>>>>>>>> }
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> To summarize, I'll need to write another patch which replaces
>>>>>>>>>>>>> Util.readURLAsBytes implementation via AOP and prints IOException
>>>>>>>>>>>>> details.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> #
>>>>>>>>>>>>>>>>>>>> CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml
>>>>>>>>>>>>>>>>>>>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M
>>>>>>>>>>>>>>>>>>>> -XX:MaxPermSize=1024m
>>>>>>>>>>>>>>>>>>>> -Dgwt.compiler.localWorkers=1"
>>>>>>>>>>>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}"
>>>>>>>>>>>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D
>>>>>>>>>>>>>>>>>>>> "ovirt_build_extra_flags
>>>>>>>>>>>>>>>>>>>> -gs
>>>>>>>>>>>>>>>>>>>> ${CI_MAVEN_SETTINGS}
>>>>>>>>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera
>>>>>>>>>>>>>>>>>>>> -Dgwt.logLevel=TRACE -X" --rebuild
>>>>>>>>>>>>>>>>>>>> rpmbuild/SRPMS/*.src.rpm
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Let me know if you need anything else.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Vojtech
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>>>>>>>>>>> From: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>>>>>>>>>>>>>>>> To: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>>>>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>>>>>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko"
>>>>>>>>>>>>>>>>>>>>>> <knesenko at redhat.com>,
>>>>>>>>>>>>>>>>>>>>>> "arch" <arch at ovirt.org>
>>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM
>>>>>>>>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> In your log I see following error:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>>>>>>> [INFO] Computing all possible rebind results for
>>>>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
>>>>>>>>>>>>>>>>>>>>>> [INFO] Rebinding
>>>>>>>>>>>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
>>>>>>>>>>>>>>>>>>>>>> [INFO] Invoking generator
>>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
>>>>>>>>>>>>>>>>>>>>>> [INFO] [ERROR] Generator
>>>>>>>>>>>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
>>>>>>>>>>>>>>>>>>>>>> threw
>>>>>>>>>>>>>>>>>>>>>> an exception while rebinding
>>>>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
>>>>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever
>>>>>>>>>>>>>>>>>>>>>> computed
>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>> removeBrickCommitRequired
>>>>>>>>>>>>>>>>>>>>>> [INFO] at
>>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
>>>>>>>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I checked again logs at
>>>>>>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host/consoleFull
>>>>>>>>>>>>>>>>>>>>>> and indeed there is a similar error:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>>>>>>> [INFO] Computing all possible rebind results for
>>>>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
>>>>>>>>>>>>>>>>>>>>>> [INFO] Rebinding
>>>>>>>>>>>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
>>>>>>>>>>>>>>>>>>>>>> [INFO] Invoking generator
>>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
>>>>>>>>>>>>>>>>>>>>>> [INFO] [ERROR] Generator
>>>>>>>>>>>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
>>>>>>>>>>>>>>>>>>>>>> threw
>>>>>>>>>>>>>>>>>>>>>> an exception while rebinding
>>>>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
>>>>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException
>>>>>>>>>>>>>>>>>>>>>> [INFO] at
>>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
>>>>>>>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> NullPointerException when processing
>>>>>>>>>>>>>>>>>>>>>> ApplicationResources
>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>> occuring
>>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>> different places (ImageResourceGenerator vs.
>>>>>>>>>>>>>>>>>>>>>> InlineResourceContext)
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Now I understand why Maven build fails without any
>>>>>>>>>>>>>>>>>>>>>> details:
>>>>>>>>>>>>>>>>>>>>>> - GWT deferred binding fails for ApplicationResources
>>>>>>>>>>>>>>>>>>>>>> (i.e.
>>>>>>>>>>>>>>>>>>>>>> uncaught
>>>>>>>>>>>>>>>>>>>>>> NPE)
>>>>>>>>>>>>>>>>>>>>>> - GWT compilation proceeds, fails to link with
>>>>>>>>>>>>>>>>>>>>>> ApplicationResources
>>>>>>>>>>>>>>>>>>>>>> - GWT compilation task fails with non-OK status code
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> So the issue here is GWT deferred binding generator
>>>>>>>>>>>>>>>>>>>>>> (InlineClientBundleGenerator) failure during GWT
>>>>>>>>>>>>>>>>>>>>>> compilation.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> GWT compiler writes extra Java files during deferred
>>>>>>>>>>>>>>>>>>>>>> binding
>>>>>>>>>>>>>>>>>>>>>> phase, so this could point to issue with local
>>>>>>>>>>>>>>>>>>>>>> environment..
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I will try to investigate this and look at GWT source
>>>>>>>>>>>>>>>>>>>>>> code.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Sandro Bonazzola
>>>>>>>>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community
>>>>>>>>>>>>>>>>>>>> collaboration.
>>>>>>>>>>>>>>>>>>>> See how it works at redhat.com
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Sandro Bonazzola
>>>>>>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community
>>>>>>>>>>>>>>>>>> collaboration.
>>>>>>>>>>>>>>>>>> See how it works at redhat.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Sandro Bonazzola
>>>>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community
>>>>>>>>>>>>>>>> collaboration.
>>>>>>>>>>>>>>>> See how it works at redhat.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Sandro Bonazzola
>>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community
>>>>>>>>>>>>>> collaboration.
>>>>>>>>>>>>>> See how it works at redhat.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Sandro Bonazzola
>>>>>>>>>>>> Better technology. Faster innovation. Powered by community
>>>>>>>>>>>> collaboration.
>>>>>>>>>>>> See how it works at redhat.com
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Sandro Bonazzola
>>>>>>>>> Better technology. Faster innovation. Powered by community
>>>>>>>>> collaboration.
>>>>>>>>> See how it works at redhat.com
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Arch mailing list
>>>>>>>> Arch at ovirt.org
>>>>>>>> http://lists.ovirt.org/mailman/listinfo/arch
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Sandro Bonazzola
>>>>>> Better technology. Faster innovation. Powered by community
>>>>>> collaboration.
>>>>>> See how it works at redhat.com
>>>>>> _______________________________________________
>>>>>> Arch mailing list
>>>>>> Arch at ovirt.org
>>>>>> http://lists.ovirt.org/mailman/listinfo/arch
>>>>>>
>>>>
>>>>
>>>> --
>>>> Sandro Bonazzola
>>>> Better technology. Faster innovation. Powered by community collaboration.
>>>> See how it works at redhat.com
>>>>
>>
>>
>> --
>> Sandro Bonazzola
>> Better technology. Faster innovation. Powered by community collaboration.
>> See how it works at redhat.com
>>
--
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com
More information about the Arch
mailing list