Il 24/02/2014 09:08, Sandro Bonazzola ha scritto:
Top posting,
Please give this task really urgent priority, as you can see:
http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/ and http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/
are failing consistently so we don't have nightly builds.
CCing infra.
Il 20/02/2014 13:34, Vojtech Szocs ha scritto:
----- Original Message -----
From: "Sandro Bonazzola" <sbonazzo@redhat.com> To: "Vojtech Szocs" <vszocs@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "Juan Hernandez" <jhernand@redhat.com>, "Kiril Nesenko" <knesenko@redhat.com>, "arch" <arch@ovirt.org> Sent: Thursday, February 20, 2014 10:02:00 AM Subject: Re: ovirt-engine build failure
Il 19/02/2014 16:51, Vojtech Szocs ha scritto:
----- Original Message -----
From: "Sandro Bonazzola" <sbonazzo@redhat.com> To: "Vojtech Szocs" <vszocs@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "Juan Hernandez" <jhernand@redhat.com>, "Kiril Nesenko" <knesenko@redhat.com>, "arch" <arch@ovirt.org> Sent: Wednesday, February 19, 2014 2:43:44 PM Subject: Re: ovirt-engine build failure
Il 19/02/2014 14:41, Vojtech Szocs ha scritto:
----- Original Message ----- > From: "Sandro Bonazzola" <sbonazzo@redhat.com> > To: "Vojtech Szocs" <vszocs@redhat.com> > Cc: "Eyal Edri" <eedri@redhat.com>, "Juan Hernandez" > <jhernand@redhat.com>, "Kiril Nesenko" <knesenko@redhat.com>, > "arch" <arch@ovirt.org> > Sent: Wednesday, February 19, 2014 2:23:16 PM > Subject: Re: ovirt-engine build failure > > Il 19/02/2014 14:19, Vojtech Szocs ha scritto: >> >> >> ----- Original Message ----- >>> From: "Sandro Bonazzola" <sbonazzo@redhat.com> >>> To: "Vojtech Szocs" <vszocs@redhat.com> >>> Cc: "Eyal Edri" <eedri@redhat.com>, "Juan Hernandez" >>> <jhernand@redhat.com>, "Kiril Nesenko" <knesenko@redhat.com>, >>> "arch" <arch@ovirt.org> >>> Sent: Wednesday, February 19, 2014 1:18:28 PM >>> Subject: Re: ovirt-engine build failure >>> >>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto: >>>> Hi, I checked GWT sources: >>>> >>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed >>>>> for >>>>> removeBrickCommitRequired >>>>> [INFO] at >>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475) >>>> >>>> This happened because internal image cache for bundled (composite >>>> PNG) >>>> images was corrput. >>>> It seems to be caused by disk I/O error while reading file >>>> images/glusterAsyncTasks/remove_brick_commit_required.png >>>> >>>>> [INFO] java.lang.NullPointerException >>>>> [INFO] at >>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40) >>>> >>>> This happened because image data was null, again possible disk I/O >>>> error >>>> while reading image file. >>>> >>>> Can we enable "TRACE" logging during GWT compilation? >>>> >>>> -Dgwt.logLevel=TRACE >>> >>> >>> Building master right now with your suggestions here: >>> http://jenkins.ovirt.org/job/manual-build-tarball/227/ >> >> Looking at >> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host/co... >> >> So far I see no errors.. There should be total 5 browsers x 8 locales = >> 40 >> GWT permutations. > > So no changes with and without > -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera. > We've still 40 permutations. Maybe you defined > userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the POM files?
You are right. I checked frontend/webadmin/modules/pom.xml -> the "gwt.userAgent" property is copied into "locale" GWT property in {WebAdmin,UserPortal}.gwt.xml
Ok, so I'll remove -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera from build command.
Agreed.
> > This: > http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/con... > failed, you can look at it, it has -Dgwt.logLevel=TRACE -X.
Hm, it contains:
[DEBUG] (f) logLevel = INFO
Somehow the TRACE log level for GWT wasn't applied.. not sure why. Let me check this on local build.
Ok, let me know if I've to change command line.
I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/
When it's merged, setting "gwt.logLevel" will work (as you did before) and we'll be able to see more details in the build log.
Here is the new build with your patch merged:
http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/con...
Thanks. I did some more investigation on this.
Again it's NullPointerException when rebinding ApplicationResources, but this time for different image file: ApplicationResources.wrenchImage ("images/wrench.png").
Looking at stacktrace & GWT sources, the problematic code seems to be (AbstractResourceContext:74):
byte[] bytes = Util.readURLAsBytes(resource);
where "resource" is (ImageResourceGenerator:311):
localized.getUrl(); // LocalizedImage localized
Note: in oVirt UI we don't use localized images (i.e. "wrench_fr_FR.png") so "localized" in this case points to "wrench.png".
So the problem seems to be reading this file:
frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png
Since we're having many GWT permutations, this file ^^ (along with similar ones) is attempted to be read multiple times *from the same (single) GWT compiler Java process, before actual permutation worker Java sub-processes are spawned*. (More precisely, above NullPointerException happens at deferred binding phase which precedes actual GWT compilation per each permutation.)
I think the root cause is Util.readURLAsBytes method used internally by GWT compiler:
public static byte[] readURLAsBytes(URL url) { try { URLConnection conn = url.openConnection(); conn.setUseCaches(false); return readURLConnectionAsBytes(conn); } catch (IOException e) { return null; // This causes NullPointerException, but this exception isn't logged (wtf, GWT!) } }
To summarize, I'll need to write another patch which replaces Util.readURLAsBytes implementation via AOP and prints IOException details.
> > > > >> >>> >>> # CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml >>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M -XX:MaxPermSize=1024m >>> -Dgwt.compiler.localWorkers=1" >>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}" >>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D "ovirt_build_extra_flags >>> -gs >>> ${CI_MAVEN_SETTINGS} -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera >>> -Dgwt.logLevel=TRACE -X" --rebuild rpmbuild/SRPMS/*.src.rpm >>> >>> Let me know if you need anything else. >>> >>> >>>> >>>> Vojtech >>>> >>>> >>>> ----- Original Message ----- >>>>> From: "Vojtech Szocs" <vszocs@redhat.com> >>>>> To: "Sandro Bonazzola" <sbonazzo@redhat.com> >>>>> Cc: "Eyal Edri" <eedri@redhat.com>, "Juan Hernandez" >>>>> <jhernand@redhat.com>, "Kiril Nesenko" <knesenko@redhat.com>, >>>>> "arch" <arch@ovirt.org> >>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM >>>>> Subject: Re: ovirt-engine build failure >>>>> >>>>> In your log I see following error: >>>>> >>>>> ... >>>>> [INFO] Computing all possible rebind results for >>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>> [INFO] Rebinding >>>>> org.ovirt.engine.ui.webadmin.ApplicationResources >>>>> [INFO] Invoking generator >>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator >>>>> [INFO] [ERROR] Generator >>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator' >>>>> threw >>>>> an exception while rebinding >>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed >>>>> for >>>>> removeBrickCommitRequired >>>>> [INFO] at >>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475) >>>>> ... >>>>> >>>>> I checked again logs at >>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host/co... >>>>> and indeed there is a similar error: >>>>> >>>>> ... >>>>> [INFO] Computing all possible rebind results for >>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>> [INFO] Rebinding >>>>> org.ovirt.engine.ui.webadmin.ApplicationResources >>>>> [INFO] Invoking generator >>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator >>>>> [INFO] [ERROR] Generator >>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator' >>>>> threw >>>>> an exception while rebinding >>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>> [INFO] java.lang.NullPointerException >>>>> [INFO] at >>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40) >>>>> ... >>>>> >>>>> NullPointerException when processing ApplicationResources but >>>>> occuring >>>>> from >>>>> different places (ImageResourceGenerator vs. InlineResourceContext) >>>>> >>>>> Now I understand why Maven build fails without any details: >>>>> - GWT deferred binding fails for ApplicationResources (i.e. uncaught >>>>> NPE) >>>>> - GWT compilation proceeds, fails to link with ApplicationResources >>>>> - GWT compilation task fails with non-OK status code >>>>> >>>>> So the issue here is GWT deferred binding generator >>>>> (InlineClientBundleGenerator) failure during GWT compilation. >>>>> >>>>> GWT compiler writes extra Java files during deferred binding >>>>> phase, so this could point to issue with local environment.. >>>>> >>>>> I will try to investigate this and look at GWT source code. >>> >>> >>> -- >>> Sandro Bonazzola >>> Better technology. Faster innovation. Powered by community >>> collaboration. >>> See how it works at redhat.com >>> > > > -- > Sandro Bonazzola > Better technology. Faster innovation. Powered by community > collaboration. > See how it works at redhat.com >
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com