[urgent] Re: ovirt-engine build failure
Vojtech Szocs
vszocs at redhat.com
Tue Feb 25 12:05:35 UTC 2014
----- Original Message -----
> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> To: "Vojtech Szocs" <vszocs at redhat.com>
> Cc: "Juan Hernandez" <jhernand at redhat.com>, "arch" <arch at ovirt.org>, "Kiril Nesenko" <knesenko at redhat.com>
> Sent: Tuesday, February 25, 2014 9:02:11 AM
> Subject: Re: [urgent] Re: ovirt-engine build failure
>
> Il 25/02/2014 08:47, Sandro Bonazzola ha scritto:
> > Il 24/02/2014 18:28, Vojtech Szocs ha scritto:
> >>
> >>
> >> ----- Original Message -----
> >>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> >>> "arch" <arch at ovirt.org>
> >>> Sent: Monday, February 24, 2014 9:08:09 AM
> >>> Subject: [urgent] Re: ovirt-engine build failure
> >>>
> >>> Top posting,
> >>>
> >>> Please give this task really urgent priority, as you can see:
> >>>
> >>> http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/
> >>> and
> >>> http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/
> >>>
> >>> are failing consistently so we don't have nightly builds.
> >>
> >> (CC'ing Einav)
> >>
> >> Attached TEST patch, please apply & rebuild ovirt-engine as usual.
> >>
> >> This patch should cause following output during GWT compilation:
> >>
> >> [ERROR] ######### Util.readURLAsBytes
> >> file:/home/vszocs/work/ovirt-engine/frontend/webadmin/modules/webadmin/target/generated-sources/gwt/org/ovirt/engine/ui/webadmin/images/search_button.png
> >>
> >> If IOException occurs, exception stacktrace will be printed right below
> >> that line.
> >>
> >> Note: please ignore the [ERROR] prefix, it's a technical issue, more
> >> details at
> >> https://www.mail-archive.com/engine-devel@ovirt.org/msg05735.html
> >>
> >> I just did a local upstream build with TEST patch applied, GWT compilation
> >> passed OK.
> >
> > build job running with your patch here:
> > http://jenkins.ovirt.org/job/manual-build-tarball/235/
> > waiting for a free executor right now.
> > Let's see how it goes.
>
> Failed on CentOS:
> http://jenkins.ovirt.org/job/manual-build-tarball/235/label=centos6-host/consoleFull
Hm, in that ^^ build, following exceptions occured:
1.
[INFO] java.lang.NullPointerException
[INFO] at com.google.gwt.dev.util.Util.computeStrongName(Util.java:172)
2.
[INFO] java.lang.NullPointerException: No ImageRect ever computed for downDisabledImage
[INFO] at com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475
1. is caused by Util#readURLAsBytes but I don't see any IOException in the logs
2. is just follow-up consequence of 1. (i.e. image metadata missing in GWT image cache)
Now I noticed I missed one possible IOException inside Util#readURLConnectionAsBytes ...
Long story short, I'll send updated TEST patch soon, apologies for the inconvenience.
>
>
>
>
>
>
> >
> >
> >
> >>
> >>>
> >>>
> >>>
> >>>
> >>> Il 20/02/2014 13:34, Vojtech Szocs ha scritto:
> >>>>
> >>>>
> >>>> ----- Original Message -----
> >>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> >>>>> "arch" <arch at ovirt.org>
> >>>>> Sent: Thursday, February 20, 2014 10:02:00 AM
> >>>>> Subject: Re: ovirt-engine build failure
> >>>>>
> >>>>> Il 19/02/2014 16:51, Vojtech Szocs ha scritto:
> >>>>>>
> >>>>>>
> >>>>>> ----- Original Message -----
> >>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> >>>>>>> "arch" <arch at ovirt.org>
> >>>>>>> Sent: Wednesday, February 19, 2014 2:43:44 PM
> >>>>>>> Subject: Re: ovirt-engine build failure
> >>>>>>>
> >>>>>>> Il 19/02/2014 14:41, Vojtech Szocs ha scritto:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ----- Original Message -----
> >>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> >>>>>>>>> "arch" <arch at ovirt.org>
> >>>>>>>>> Sent: Wednesday, February 19, 2014 2:23:16 PM
> >>>>>>>>> Subject: Re: ovirt-engine build failure
> >>>>>>>>>
> >>>>>>>>> Il 19/02/2014 14:19, Vojtech Szocs ha scritto:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> >>>>>>>>>>> "arch" <arch at ovirt.org>
> >>>>>>>>>>> Sent: Wednesday, February 19, 2014 1:18:28 PM
> >>>>>>>>>>> Subject: Re: ovirt-engine build failure
> >>>>>>>>>>>
> >>>>>>>>>>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto:
> >>>>>>>>>>>> Hi, I checked GWT sources:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever
> >>>>>>>>>>>>> computed
> >>>>>>>>>>>>> for
> >>>>>>>>>>>>> removeBrickCommitRequired
> >>>>>>>>>>>>> [INFO] at
> >>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
> >>>>>>>>>>>>
> >>>>>>>>>>>> This happened because internal image cache for bundled
> >>>>>>>>>>>> (composite
> >>>>>>>>>>>> PNG)
> >>>>>>>>>>>> images was corrput.
> >>>>>>>>>>>> It seems to be caused by disk I/O error while reading file
> >>>>>>>>>>>> images/glusterAsyncTasks/remove_brick_commit_required.png
> >>>>>>>>>>>>
> >>>>>>>>>>>>> [INFO] java.lang.NullPointerException
> >>>>>>>>>>>>> [INFO] at
> >>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
> >>>>>>>>>>>>
> >>>>>>>>>>>> This happened because image data was null, again possible disk
> >>>>>>>>>>>> I/O
> >>>>>>>>>>>> error
> >>>>>>>>>>>> while reading image file.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Can we enable "TRACE" logging during GWT compilation?
> >>>>>>>>>>>>
> >>>>>>>>>>>> -Dgwt.logLevel=TRACE
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Building master right now with your suggestions here:
> >>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/
> >>>>>>>>>>
> >>>>>>>>>> Looking at
> >>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host/console
> >>>>>>>>>>
> >>>>>>>>>> So far I see no errors.. There should be total 5 browsers x 8
> >>>>>>>>>> locales
> >>>>>>>>>> =
> >>>>>>>>>> 40
> >>>>>>>>>> GWT permutations.
> >>>>>>>>>
> >>>>>>>>> So no changes with and without
> >>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera.
> >>>>>>>>> We've still 40 permutations. Maybe you defined
> >>>>>>>>> userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the POM files?
> >>>>>>>>
> >>>>>>>> You are right. I checked frontend/webadmin/modules/pom.xml -> the
> >>>>>>>> "gwt.userAgent"
> >>>>>>>> property is copied into "locale" GWT property in
> >>>>>>>> {WebAdmin,UserPortal}.gwt.xml
> >>>>>>>
> >>>>>>> Ok, so I'll remove -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera from
> >>>>>>> build
> >>>>>>> command.
> >>>>>>
> >>>>>> Agreed.
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> This:
> >>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/console
> >>>>>>>>> failed, you can look at it, it has -Dgwt.logLevel=TRACE -X.
> >>>>>>>>
> >>>>>>>> Hm, it contains:
> >>>>>>>>
> >>>>>>>> [DEBUG] (f) logLevel = INFO
> >>>>>>>>
> >>>>>>>> Somehow the TRACE log level for GWT wasn't applied.. not sure why.
> >>>>>>>> Let
> >>>>>>>> me
> >>>>>>>> check
> >>>>>>>> this on local build.
> >>>>>>>
> >>>>>>> Ok, let me know if I've to change command line.
> >>>>>>
> >>>>>> I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/
> >>>>>>
> >>>>>> When it's merged, setting "gwt.logLevel" will work (as you did before)
> >>>>>> and we'll be able to see more details in the build log.
> >>>>>
> >>>>> Here is the new build with your patch merged:
> >>>>>
> >>>>> http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/consoleFull
> >>>>
> >>>> Thanks. I did some more investigation on this.
> >>>>
> >>>> Again it's NullPointerException when rebinding ApplicationResources, but
> >>>> this time for different image file: ApplicationResources.wrenchImage
> >>>> ("images/wrench.png").
> >>>>
> >>>> Looking at stacktrace & GWT sources, the problematic code seems to be
> >>>> (AbstractResourceContext:74):
> >>>>
> >>>> byte[] bytes = Util.readURLAsBytes(resource);
> >>>>
> >>>> where "resource" is (ImageResourceGenerator:311):
> >>>>
> >>>> localized.getUrl(); // LocalizedImage localized
> >>>>
> >>>> Note: in oVirt UI we don't use localized images (i.e.
> >>>> "wrench_fr_FR.png")
> >>>> so "localized" in this case points to "wrench.png".
> >>>>
> >>>> So the problem seems to be reading this file:
> >>>>
> >>>> frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png
> >>>>
> >>>> Since we're having many GWT permutations, this file ^^ (along with
> >>>> similar
> >>>> ones) is attempted to be read multiple times *from the same (single) GWT
> >>>> compiler Java process, before actual permutation worker Java
> >>>> sub-processes
> >>>> are spawned*. (More precisely, above NullPointerException happens at
> >>>> deferred binding phase which precedes actual GWT compilation per each
> >>>> permutation.)
> >>>>
> >>>> I think the root cause is Util.readURLAsBytes method used internally by
> >>>> GWT
> >>>> compiler:
> >>>>
> >>>> public static byte[] readURLAsBytes(URL url) {
> >>>> try {
> >>>> URLConnection conn = url.openConnection();
> >>>> conn.setUseCaches(false);
> >>>> return readURLConnectionAsBytes(conn);
> >>>> } catch (IOException e) {
> >>>> return null; // This causes NullPointerException, but this
> >>>> exception
> >>>> isn't logged (wtf, GWT!)
> >>>> }
> >>>> }
> >>>>
> >>>> To summarize, I'll need to write another patch which replaces
> >>>> Util.readURLAsBytes implementation via AOP and prints IOException
> >>>> details.
> >>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> # CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml
> >>>>>>>>>>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M
> >>>>>>>>>>> -XX:MaxPermSize=1024m
> >>>>>>>>>>> -Dgwt.compiler.localWorkers=1"
> >>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}"
> >>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D
> >>>>>>>>>>> "ovirt_build_extra_flags
> >>>>>>>>>>> -gs
> >>>>>>>>>>> ${CI_MAVEN_SETTINGS}
> >>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera
> >>>>>>>>>>> -Dgwt.logLevel=TRACE -X" --rebuild rpmbuild/SRPMS/*.src.rpm
> >>>>>>>>>>>
> >>>>>>>>>>> Let me know if you need anything else.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Vojtech
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>> From: "Vojtech Szocs" <vszocs at redhat.com>
> >>>>>>>>>>>>> To: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> >>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> >>>>>>>>>>>>> "arch" <arch at ovirt.org>
> >>>>>>>>>>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM
> >>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> In your log I see following error:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ...
> >>>>>>>>>>>>> [INFO] Computing all possible rebind results for
> >>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> >>>>>>>>>>>>> [INFO] Rebinding
> >>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
> >>>>>>>>>>>>> [INFO] Invoking generator
> >>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
> >>>>>>>>>>>>> [INFO] [ERROR] Generator
> >>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
> >>>>>>>>>>>>> threw
> >>>>>>>>>>>>> an exception while rebinding
> >>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> >>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever
> >>>>>>>>>>>>> computed
> >>>>>>>>>>>>> for
> >>>>>>>>>>>>> removeBrickCommitRequired
> >>>>>>>>>>>>> [INFO] at
> >>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
> >>>>>>>>>>>>> ...
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I checked again logs at
> >>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host/consoleFull
> >>>>>>>>>>>>> and indeed there is a similar error:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ...
> >>>>>>>>>>>>> [INFO] Computing all possible rebind results for
> >>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> >>>>>>>>>>>>> [INFO] Rebinding
> >>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
> >>>>>>>>>>>>> [INFO] Invoking generator
> >>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
> >>>>>>>>>>>>> [INFO] [ERROR] Generator
> >>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
> >>>>>>>>>>>>> threw
> >>>>>>>>>>>>> an exception while rebinding
> >>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> >>>>>>>>>>>>> [INFO] java.lang.NullPointerException
> >>>>>>>>>>>>> [INFO] at
> >>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
> >>>>>>>>>>>>> ...
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> NullPointerException when processing ApplicationResources but
> >>>>>>>>>>>>> occuring
> >>>>>>>>>>>>> from
> >>>>>>>>>>>>> different places (ImageResourceGenerator vs.
> >>>>>>>>>>>>> InlineResourceContext)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Now I understand why Maven build fails without any details:
> >>>>>>>>>>>>> - GWT deferred binding fails for ApplicationResources (i.e.
> >>>>>>>>>>>>> uncaught
> >>>>>>>>>>>>> NPE)
> >>>>>>>>>>>>> - GWT compilation proceeds, fails to link with
> >>>>>>>>>>>>> ApplicationResources
> >>>>>>>>>>>>> - GWT compilation task fails with non-OK status code
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> So the issue here is GWT deferred binding generator
> >>>>>>>>>>>>> (InlineClientBundleGenerator) failure during GWT compilation.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> GWT compiler writes extra Java files during deferred binding
> >>>>>>>>>>>>> phase, so this could point to issue with local environment..
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I will try to investigate this and look at GWT source code.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> Sandro Bonazzola
> >>>>>>>>>>> Better technology. Faster innovation. Powered by community
> >>>>>>>>>>> collaboration.
> >>>>>>>>>>> See how it works at redhat.com
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Sandro Bonazzola
> >>>>>>>>> Better technology. Faster innovation. Powered by community
> >>>>>>>>> collaboration.
> >>>>>>>>> See how it works at redhat.com
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Sandro Bonazzola
> >>>>>>> Better technology. Faster innovation. Powered by community
> >>>>>>> collaboration.
> >>>>>>> See how it works at redhat.com
> >>>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Sandro Bonazzola
> >>>>> Better technology. Faster innovation. Powered by community
> >>>>> collaboration.
> >>>>> See how it works at redhat.com
> >>>>>
> >>>
> >>>
> >>> --
> >>> Sandro Bonazzola
> >>> Better technology. Faster innovation. Powered by community collaboration.
> >>> See how it works at redhat.com
> >>>
> >
> >
>
>
> --
> Sandro Bonazzola
> Better technology. Faster innovation. Powered by community collaboration.
> See how it works at redhat.com
>
More information about the Arch
mailing list