[urgent] Re: ovirt-engine build failure
Vojtech Szocs
vszocs at redhat.com
Mon Feb 24 17:28:26 UTC 2014
----- Original Message -----
> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> To: "Vojtech Szocs" <vszocs at redhat.com>
> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez" <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> "arch" <arch at ovirt.org>
> Sent: Monday, February 24, 2014 9:08:09 AM
> Subject: [urgent] Re: ovirt-engine build failure
>
> Top posting,
>
> Please give this task really urgent priority, as you can see:
>
> http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/
> and
> http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/
>
> are failing consistently so we don't have nightly builds.
(CC'ing Einav)
Attached TEST patch, please apply & rebuild ovirt-engine as usual.
This patch should cause following output during GWT compilation:
[ERROR] ######### Util.readURLAsBytes file:/home/vszocs/work/ovirt-engine/frontend/webadmin/modules/webadmin/target/generated-sources/gwt/org/ovirt/engine/ui/webadmin/images/search_button.png
If IOException occurs, exception stacktrace will be printed right below that line.
Note: please ignore the [ERROR] prefix, it's a technical issue, more details at https://www.mail-archive.com/engine-devel@ovirt.org/msg05735.html
I just did a local upstream build with TEST patch applied, GWT compilation passed OK.
>
>
>
>
> Il 20/02/2014 13:34, Vojtech Szocs ha scritto:
> >
> >
> > ----- Original Message -----
> >> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >> To: "Vojtech Szocs" <vszocs at redhat.com>
> >> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> >> "arch" <arch at ovirt.org>
> >> Sent: Thursday, February 20, 2014 10:02:00 AM
> >> Subject: Re: ovirt-engine build failure
> >>
> >> Il 19/02/2014 16:51, Vojtech Szocs ha scritto:
> >>>
> >>>
> >>> ----- Original Message -----
> >>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> >>>> "arch" <arch at ovirt.org>
> >>>> Sent: Wednesday, February 19, 2014 2:43:44 PM
> >>>> Subject: Re: ovirt-engine build failure
> >>>>
> >>>> Il 19/02/2014 14:41, Vojtech Szocs ha scritto:
> >>>>>
> >>>>>
> >>>>> ----- Original Message -----
> >>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> >>>>>> "arch" <arch at ovirt.org>
> >>>>>> Sent: Wednesday, February 19, 2014 2:23:16 PM
> >>>>>> Subject: Re: ovirt-engine build failure
> >>>>>>
> >>>>>> Il 19/02/2014 14:19, Vojtech Szocs ha scritto:
> >>>>>>>
> >>>>>>>
> >>>>>>> ----- Original Message -----
> >>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> >>>>>>>> "arch" <arch at ovirt.org>
> >>>>>>>> Sent: Wednesday, February 19, 2014 1:18:28 PM
> >>>>>>>> Subject: Re: ovirt-engine build failure
> >>>>>>>>
> >>>>>>>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto:
> >>>>>>>>> Hi, I checked GWT sources:
> >>>>>>>>>
> >>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed
> >>>>>>>>>> for
> >>>>>>>>>> removeBrickCommitRequired
> >>>>>>>>>> [INFO] at
> >>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
> >>>>>>>>>
> >>>>>>>>> This happened because internal image cache for bundled (composite
> >>>>>>>>> PNG)
> >>>>>>>>> images was corrput.
> >>>>>>>>> It seems to be caused by disk I/O error while reading file
> >>>>>>>>> images/glusterAsyncTasks/remove_brick_commit_required.png
> >>>>>>>>>
> >>>>>>>>>> [INFO] java.lang.NullPointerException
> >>>>>>>>>> [INFO] at
> >>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
> >>>>>>>>>
> >>>>>>>>> This happened because image data was null, again possible disk I/O
> >>>>>>>>> error
> >>>>>>>>> while reading image file.
> >>>>>>>>>
> >>>>>>>>> Can we enable "TRACE" logging during GWT compilation?
> >>>>>>>>>
> >>>>>>>>> -Dgwt.logLevel=TRACE
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Building master right now with your suggestions here:
> >>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/
> >>>>>>>
> >>>>>>> Looking at
> >>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host/console
> >>>>>>>
> >>>>>>> So far I see no errors.. There should be total 5 browsers x 8 locales
> >>>>>>> =
> >>>>>>> 40
> >>>>>>> GWT permutations.
> >>>>>>
> >>>>>> So no changes with and without
> >>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera.
> >>>>>> We've still 40 permutations. Maybe you defined
> >>>>>> userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the POM files?
> >>>>>
> >>>>> You are right. I checked frontend/webadmin/modules/pom.xml -> the
> >>>>> "gwt.userAgent"
> >>>>> property is copied into "locale" GWT property in
> >>>>> {WebAdmin,UserPortal}.gwt.xml
> >>>>
> >>>> Ok, so I'll remove -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera from
> >>>> build
> >>>> command.
> >>>
> >>> Agreed.
> >>>
> >>>>
> >>>>
> >>>>>
> >>>>>>
> >>>>>> This:
> >>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/console
> >>>>>> failed, you can look at it, it has -Dgwt.logLevel=TRACE -X.
> >>>>>
> >>>>> Hm, it contains:
> >>>>>
> >>>>> [DEBUG] (f) logLevel = INFO
> >>>>>
> >>>>> Somehow the TRACE log level for GWT wasn't applied.. not sure why. Let
> >>>>> me
> >>>>> check
> >>>>> this on local build.
> >>>>
> >>>> Ok, let me know if I've to change command line.
> >>>
> >>> I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/
> >>>
> >>> When it's merged, setting "gwt.logLevel" will work (as you did before)
> >>> and we'll be able to see more details in the build log.
> >>
> >> Here is the new build with your patch merged:
> >>
> >> http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/consoleFull
> >
> > Thanks. I did some more investigation on this.
> >
> > Again it's NullPointerException when rebinding ApplicationResources, but
> > this time for different image file: ApplicationResources.wrenchImage
> > ("images/wrench.png").
> >
> > Looking at stacktrace & GWT sources, the problematic code seems to be
> > (AbstractResourceContext:74):
> >
> > byte[] bytes = Util.readURLAsBytes(resource);
> >
> > where "resource" is (ImageResourceGenerator:311):
> >
> > localized.getUrl(); // LocalizedImage localized
> >
> > Note: in oVirt UI we don't use localized images (i.e. "wrench_fr_FR.png")
> > so "localized" in this case points to "wrench.png".
> >
> > So the problem seems to be reading this file:
> >
> > frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png
> >
> > Since we're having many GWT permutations, this file ^^ (along with similar
> > ones) is attempted to be read multiple times *from the same (single) GWT
> > compiler Java process, before actual permutation worker Java sub-processes
> > are spawned*. (More precisely, above NullPointerException happens at
> > deferred binding phase which precedes actual GWT compilation per each
> > permutation.)
> >
> > I think the root cause is Util.readURLAsBytes method used internally by GWT
> > compiler:
> >
> > public static byte[] readURLAsBytes(URL url) {
> > try {
> > URLConnection conn = url.openConnection();
> > conn.setUseCaches(false);
> > return readURLConnectionAsBytes(conn);
> > } catch (IOException e) {
> > return null; // This causes NullPointerException, but this exception
> > isn't logged (wtf, GWT!)
> > }
> > }
> >
> > To summarize, I'll need to write another patch which replaces
> > Util.readURLAsBytes implementation via AOP and prints IOException details.
> >
> >>
> >>
> >>
> >>
> >>>
> >>>>
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> # CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml
> >>>>>>>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M -XX:MaxPermSize=1024m
> >>>>>>>> -Dgwt.compiler.localWorkers=1"
> >>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}"
> >>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D "ovirt_build_extra_flags
> >>>>>>>> -gs
> >>>>>>>> ${CI_MAVEN_SETTINGS} -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera
> >>>>>>>> -Dgwt.logLevel=TRACE -X" --rebuild rpmbuild/SRPMS/*.src.rpm
> >>>>>>>>
> >>>>>>>> Let me know if you need anything else.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Vojtech
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ----- Original Message -----
> >>>>>>>>>> From: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>>>>> To: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> >>>>>>>>>> "arch" <arch at ovirt.org>
> >>>>>>>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM
> >>>>>>>>>> Subject: Re: ovirt-engine build failure
> >>>>>>>>>>
> >>>>>>>>>> In your log I see following error:
> >>>>>>>>>>
> >>>>>>>>>> ...
> >>>>>>>>>> [INFO] Computing all possible rebind results for
> >>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> >>>>>>>>>> [INFO] Rebinding
> >>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
> >>>>>>>>>> [INFO] Invoking generator
> >>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
> >>>>>>>>>> [INFO] [ERROR] Generator
> >>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
> >>>>>>>>>> threw
> >>>>>>>>>> an exception while rebinding
> >>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> >>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed
> >>>>>>>>>> for
> >>>>>>>>>> removeBrickCommitRequired
> >>>>>>>>>> [INFO] at
> >>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
> >>>>>>>>>> ...
> >>>>>>>>>>
> >>>>>>>>>> I checked again logs at
> >>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host/consoleFull
> >>>>>>>>>> and indeed there is a similar error:
> >>>>>>>>>>
> >>>>>>>>>> ...
> >>>>>>>>>> [INFO] Computing all possible rebind results for
> >>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> >>>>>>>>>> [INFO] Rebinding
> >>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
> >>>>>>>>>> [INFO] Invoking generator
> >>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
> >>>>>>>>>> [INFO] [ERROR] Generator
> >>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
> >>>>>>>>>> threw
> >>>>>>>>>> an exception while rebinding
> >>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> >>>>>>>>>> [INFO] java.lang.NullPointerException
> >>>>>>>>>> [INFO] at
> >>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
> >>>>>>>>>> ...
> >>>>>>>>>>
> >>>>>>>>>> NullPointerException when processing ApplicationResources but
> >>>>>>>>>> occuring
> >>>>>>>>>> from
> >>>>>>>>>> different places (ImageResourceGenerator vs.
> >>>>>>>>>> InlineResourceContext)
> >>>>>>>>>>
> >>>>>>>>>> Now I understand why Maven build fails without any details:
> >>>>>>>>>> - GWT deferred binding fails for ApplicationResources (i.e.
> >>>>>>>>>> uncaught
> >>>>>>>>>> NPE)
> >>>>>>>>>> - GWT compilation proceeds, fails to link with
> >>>>>>>>>> ApplicationResources
> >>>>>>>>>> - GWT compilation task fails with non-OK status code
> >>>>>>>>>>
> >>>>>>>>>> So the issue here is GWT deferred binding generator
> >>>>>>>>>> (InlineClientBundleGenerator) failure during GWT compilation.
> >>>>>>>>>>
> >>>>>>>>>> GWT compiler writes extra Java files during deferred binding
> >>>>>>>>>> phase, so this could point to issue with local environment..
> >>>>>>>>>>
> >>>>>>>>>> I will try to investigate this and look at GWT source code.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Sandro Bonazzola
> >>>>>>>> Better technology. Faster innovation. Powered by community
> >>>>>>>> collaboration.
> >>>>>>>> See how it works at redhat.com
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Sandro Bonazzola
> >>>>>> Better technology. Faster innovation. Powered by community
> >>>>>> collaboration.
> >>>>>> See how it works at redhat.com
> >>>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Sandro Bonazzola
> >>>> Better technology. Faster innovation. Powered by community
> >>>> collaboration.
> >>>> See how it works at redhat.com
> >>>>
> >>
> >>
> >> --
> >> Sandro Bonazzola
> >> Better technology. Faster innovation. Powered by community collaboration.
> >> See how it works at redhat.com
> >>
>
>
> --
> Sandro Bonazzola
> Better technology. Faster innovation. Powered by community collaboration.
> See how it works at redhat.com
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-TEST-Completely-override-Util.readURLAsBytes-impleme.patch
Type: text/x-patch
Size: 2979 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/arch/attachments/20140224/bff6889c/attachment.bin>
More information about the Arch
mailing list