Re: [urgent] Re: ovirt-engine build failure

Top posting, Please give this task really urgent priority, as you can see: http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/ and http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/ are failing consistently so we don't have nightly builds.

Il 20/02/2014 13:34, Vojtech Szocs ha scritto: > > > ----- Original Message ----- >> From: "Sandro Bonazzola" <sbonazzo(a)redhat.com&gt; >> To: "Vojtech Szocs" <vszocs(a)redhat.com&gt; >> Cc: "Eyal Edri" <eedri(a)redhat.com&gt;, "Juan Hernandez" <jhernand(a)redhat.com&gt;, "Kiril Nesenko" <knesenko(a)redhat.com&gt;, >> "arch" <arch(a)ovirt.org&gt; >> Sent: Thursday, February 20, 2014 10:02:00 AM >> Subject: Re: ovirt-engine build failure >> >> Il 19/02/2014 16:51, Vojtech Szocs ha scritto: >>> >>> >>> ----- Original Message ----- >>>> From: "Sandro Bonazzola" <sbonazzo(a)redhat.com&gt; >>>> To: "Vojtech Szocs" <vszocs(a)redhat.com&gt; >>>> Cc: "Eyal Edri" <eedri(a)redhat.com&gt;, "Juan Hernandez" >>>> <jhernand(a)redhat.com&gt;, "Kiril Nesenko" <knesenko(a)redhat.com&gt;, >>>> "arch" <arch(a)ovirt.org&gt; >>>> Sent: Wednesday, February 19, 2014 2:43:44 PM >>>> Subject: Re: ovirt-engine build failure >>>> >>>> Il 19/02/2014 14:41, Vojtech Szocs ha scritto: >>>>> >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Sandro Bonazzola" <sbonazzo(a)redhat.com&gt; >>>>>> To: "Vojtech Szocs" <vszocs(a)redhat.com&gt; >>>>>> Cc: "Eyal Edri" <eedri(a)redhat.com&gt;, "Juan Hernandez" >>>>>> <jhernand(a)redhat.com&gt;, "Kiril Nesenko" <knesenko(a)redhat.com&gt;, >>>>>> "arch" <arch(a)ovirt.org&gt; >>>>>> Sent: Wednesday, February 19, 2014 2:23:16 PM >>>>>> Subject: Re: ovirt-engine build failure >>>>>> >>>>>> Il 19/02/2014 14:19, Vojtech Szocs ha scritto: >>>>>>> >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> From: "Sandro Bonazzola" <sbonazzo(a)redhat.com&gt; >>>>>>>> To: "Vojtech Szocs" <vszocs(a)redhat.com&gt; >>>>>>>> Cc: "Eyal Edri" <eedri(a)redhat.com&gt;, "Juan Hernandez" >>>>>>>> <jhernand(a)redhat.com&gt;, "Kiril Nesenko" <knesenko(a)redhat.com&gt;, >>>>>>>> "arch" <arch(a)ovirt.org&gt; >>>>>>>> Sent: Wednesday, February 19, 2014 1:18:28 PM >>>>>>>> Subject: Re: ovirt-engine build failure >>>>>>>> >>>>>>>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto: >>>>>>>>> Hi, I checked GWT sources: >>>>>>>>> >>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed >>>>>>>>>> for >>>>>>>>>> removeBrickCommitRequired >>>>>>>>>> [INFO] at >>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475) >>>>>>>>> >>>>>>>>> This happened because internal image cache for bundled (composite >>>>>>>>> PNG) >>>>>>>>> images was corrput. >>>>>>>>> It seems to be caused by disk I/O error while reading file >>>>>>>>> images/glusterAsyncTasks/remove_brick_commit_required.png >>>>>>>>> >>>>>>>>>> [INFO] java.lang.NullPointerException >>>>>>>>>> [INFO] at >>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40) >>>>>>>>> >>>>>>>>> This happened because image data was null, again possible disk I/O >>>>>>>>> error >>>>>>>>> while reading image file. >>>>>>>>> >>>>>>>>> Can we enable "TRACE" logging during GWT compilation? >>>>>>>>> >>>>>>>>> -Dgwt.logLevel=TRACE >>>>>>>> >>>>>>>> >>>>>>>> Building master right now with your suggestions here: >>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/ >>>>>>> >>>>>>> Looking at >>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host... >>>>>>> >>>>>>> So far I see no errors.. There should be total 5 browsers x 8 locales = >>>>>>> 40 >>>>>>> GWT permutations. >>>>>> >>>>>> So no changes with and without >>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera. >>>>>> We've still 40 permutations. Maybe you defined >>>>>> userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the POM files? >>>>> >>>>> You are right. I checked frontend/webadmin/modules/pom.xml -> the >>>>> "gwt.userAgent" >>>>> property is copied into "locale" GWT property in >>>>> {WebAdmin,UserPortal}.gwt.xml >>>> >>>> Ok, so I'll remove -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera from >>>> build >>>> command. >>> >>> Agreed. >>> >>>> >>>> >>>>> >>>>>> >>>>>> This: >>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/... >>>>>> failed, you can look at it, it has -Dgwt.logLevel=TRACE -X. >>>>> >>>>> Hm, it contains: >>>>> >>>>> [DEBUG] (f) logLevel = INFO >>>>> >>>>> Somehow the TRACE log level for GWT wasn't applied.. not sure why. Let me >>>>> check >>>>> this on local build. >>>> >>>> Ok, let me know if I've to change command line. >>> >>> I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/ >>> >>> When it's merged, setting "gwt.logLevel" will work (as you did before) >>> and we'll be able to see more details in the build log. >> >> Here is the new build with your patch merged: >> >> http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/... > > Thanks. I did some more investigation on this. > > Again it's NullPointerException when rebinding ApplicationResources, but this time for different image file: ApplicationResources.wrenchImage ("images/wrench.png"). > > Looking at stacktrace & GWT sources, the problematic code seems to be (AbstractResourceContext:74): > > byte[] bytes = Util.readURLAsBytes(resource); > > where "resource" is (ImageResourceGenerator:311): > > localized.getUrl(); // LocalizedImage localized > > Note: in oVirt UI we don't use localized images (i.e. "wrench_fr_FR.png") so "localized" in this case points to "wrench.png". > > So the problem seems to be reading this file: > > frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png > > Since we're having many GWT permutations, this file ^^ (along with similar ones) is attempted to be read multiple times *from the same (single) GWT compiler Java process, before actual permutation worker Java sub-processes are spawned*. (More precisely, above NullPointerException happens at deferred binding phase which precedes actual GWT compilation per each permutation.) > > I think the root cause is Util.readURLAsBytes method used internally by GWT compiler: > > public static byte[] readURLAsBytes(URL url) { > try { > URLConnection conn = url.openConnection(); > conn.setUseCaches(false); > return readURLConnectionAsBytes(conn); > } catch (IOException e) { > return null; // This causes NullPointerException, but this exception isn't logged (wtf, GWT!) > } > } > > To summarize, I'll need to write another patch which replaces Util.readURLAsBytes implementation via AOP and prints IOException details. > >> >> >> >> >>> >>>> >>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>>> >>>>>>>> # CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml >>>>>>>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M -XX:MaxPermSize=1024m >>>>>>>> -Dgwt.compiler.localWorkers=1" >>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}" >>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D "ovirt_build_extra_flags >>>>>>>> -gs >>>>>>>> ${CI_MAVEN_SETTINGS} -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera >>>>>>>> -Dgwt.logLevel=TRACE -X" --rebuild rpmbuild/SRPMS/*.src.rpm >>>>>>>> >>>>>>>> Let me know if you need anything else. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Vojtech >>>>>>>>> >>>>>>>>> >>>>>>>>> ----- Original Message ----- >>>>>>>>>> From: "Vojtech Szocs" <vszocs(a)redhat.com&gt; >>>>>>>>>> To: "Sandro Bonazzola" <sbonazzo(a)redhat.com&gt; >>>>>>>>>> Cc: "Eyal Edri" <eedri(a)redhat.com&gt;, "Juan Hernandez" >>>>>>>>>> <jhernand(a)redhat.com&gt;, "Kiril Nesenko" <knesenko(a)redhat.com&gt;, >>>>>>>>>> "arch" <arch(a)ovirt.org&gt; >>>>>>>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM >>>>>>>>>> Subject: Re: ovirt-engine build failure >>>>>>>>>> >>>>>>>>>> In your log I see following error: >>>>>>>>>> >>>>>>>>>> ... >>>>>>>>>> [INFO] Computing all possible rebind results for >>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>>>>>>> [INFO] Rebinding >>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources >>>>>>>>>> [INFO] Invoking generator >>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator >>>>>>>>>> [INFO] [ERROR] Generator >>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator' >>>>>>>>>> threw >>>>>>>>>> an exception while rebinding >>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed >>>>>>>>>> for >>>>>>>>>> removeBrickCommitRequired >>>>>>>>>> [INFO] at >>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475) >>>>>>>>>> ... >>>>>>>>>> >>>>>>>>>> I checked again logs at >>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host... >>>>>>>>>> and indeed there is a similar error: >>>>>>>>>> >>>>>>>>>> ... >>>>>>>>>> [INFO] Computing all possible rebind results for >>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>>>>>>> [INFO] Rebinding >>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources >>>>>>>>>> [INFO] Invoking generator >>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator >>>>>>>>>> [INFO] [ERROR] Generator >>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator' >>>>>>>>>> threw >>>>>>>>>> an exception while rebinding >>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>>>>>>> [INFO] java.lang.NullPointerException >>>>>>>>>> [INFO] at >>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40) >>>>>>>>>> ... >>>>>>>>>> >>>>>>>>>> NullPointerException when processing ApplicationResources but >>>>>>>>>> occuring >>>>>>>>>> from >>>>>>>>>> different places (ImageResourceGenerator vs. InlineResourceContext) >>>>>>>>>> >>>>>>>>>> Now I understand why Maven build fails without any details: >>>>>>>>>> - GWT deferred binding fails for ApplicationResources (i.e. uncaught >>>>>>>>>> NPE) >>>>>>>>>> - GWT compilation proceeds, fails to link with ApplicationResources >>>>>>>>>> - GWT compilation task fails with non-OK status code >>>>>>>>>> >>>>>>>>>> So the issue here is GWT deferred binding generator >>>>>>>>>> (InlineClientBundleGenerator) failure during GWT compilation. >>>>>>>>>> >>>>>>>>>> GWT compiler writes extra Java files during deferred binding >>>>>>>>>> phase, so this could point to issue with local environment.. >>>>>>>>>> >>>>>>>>>> I will try to investigate this and look at GWT source code. >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Sandro Bonazzola >>>>>>>> Better technology. Faster innovation. Powered by community >>>>>>>> collaboration. >>>>>>>> See how it works at redhat.com >>>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Sandro Bonazzola >>>>>> Better technology. Faster innovation. Powered by community >>>>>> collaboration. >>>>>> See how it works at redhat.com >>>>>> >>>> >>>> >>>> -- >>>> Sandro Bonazzola >>>> Better technology. Faster innovation. Powered by community collaboration. >>>> See how it works at redhat.com >>>> >> >> >> -- >> Sandro Bonazzola >> Better technology. Faster innovation. Powered by community collaboration. >> See how it works at redhat.com >>