[urgent] Re: ovirt-engine build failure

Vojtech Szocs vszocs at redhat.com
Tue Feb 25 15:52:22 UTC 2014 


----- Original Message -----
> From: "Vojtech Szocs" <vszocs at redhat.com>
> To: "Sandro Bonazzola" <sbonazzo at redhat.com>
> Cc: "Juan Hernandez" <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>, "arch" <arch at ovirt.org>
> Sent: Tuesday, February 25, 2014 1:05:35 PM
> Subject: Re: [urgent] Re: ovirt-engine build failure
> 
> 
> 
> ----- Original Message -----
> > From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> > To: "Vojtech Szocs" <vszocs at redhat.com>
> > Cc: "Juan Hernandez" <jhernand at redhat.com>, "arch" <arch at ovirt.org>, "Kiril
> > Nesenko" <knesenko at redhat.com>
> > Sent: Tuesday, February 25, 2014 9:02:11 AM
> > Subject: Re: [urgent] Re: ovirt-engine build failure
> > 
> > Il 25/02/2014 08:47, Sandro Bonazzola ha scritto:
> > > Il 24/02/2014 18:28, Vojtech Szocs ha scritto:
> > >>
> > >>
> > >> ----- Original Message -----
> > >>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> > >>> To: "Vojtech Szocs" <vszocs at redhat.com>
> > >>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> > >>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> > >>> "arch" <arch at ovirt.org>
> > >>> Sent: Monday, February 24, 2014 9:08:09 AM
> > >>> Subject: [urgent] Re: ovirt-engine build failure
> > >>>
> > >>> Top posting,
> > >>>
> > >>> Please give this task really urgent priority, as you can see:
> > >>>
> > >>> http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/
> > >>> and
> > >>> http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/
> > >>>
> > >>> are failing consistently so we don't have nightly builds.
> > >>
> > >> (CC'ing Einav)
> > >>
> > >> Attached TEST patch, please apply & rebuild ovirt-engine as usual.
> > >>
> > >> This patch should cause following output during GWT compilation:
> > >>
> > >> [ERROR] ######### Util.readURLAsBytes
> > >> file:/home/vszocs/work/ovirt-engine/frontend/webadmin/modules/webadmin/target/generated-sources/gwt/org/ovirt/engine/ui/webadmin/images/search_button.png
> > >>
> > >> If IOException occurs, exception stacktrace will be printed right below
> > >> that line.
> > >>
> > >> Note: please ignore the [ERROR] prefix, it's a technical issue, more
> > >> details at
> > >> https://www.mail-archive.com/engine-devel@ovirt.org/msg05735.html
> > >>
> > >> I just did a local upstream build with TEST patch applied, GWT
> > >> compilation
> > >> passed OK.
> > > 
> > > build job running with your patch here:
> > > http://jenkins.ovirt.org/job/manual-build-tarball/235/
> > > waiting for a free executor right now.
> > > Let's see how it goes.
> > 
> > Failed on CentOS:
> > http://jenkins.ovirt.org/job/manual-build-tarball/235/label=centos6-host/consoleFull
> 
> Hm, in that ^^ build, following exceptions occured:
> 
> 1.
> [INFO] java.lang.NullPointerException
> [INFO]  at com.google.gwt.dev.util.Util.computeStrongName(Util.java:172)
> 
> 2.
> [INFO] java.lang.NullPointerException: No ImageRect ever computed for
> downDisabledImage
> [INFO]  at
> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475
> 
> 1. is caused by Util#readURLAsBytes but I don't see any IOException in the
> logs
> 2. is just follow-up consequence of 1. (i.e. image metadata missing in GWT
> image cache)
> 
> Now I noticed I missed one possible IOException inside
> Util#readURLConnectionAsBytes ...
> 
> Long story short, I'll send updated TEST patch soon, apologies for the
> inconvenience.

TEST-v2 patch attached. Please apply & rebuild ovirt-engine as usual.

> 
> > 
> > 
> > 
> > 
> > 
> > 
> > > 
> > > 
> > > 
> > >>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> Il 20/02/2014 13:34, Vojtech Szocs ha scritto:
> > >>>>
> > >>>>
> > >>>> ----- Original Message -----
> > >>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> > >>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> > >>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> > >>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> > >>>>> "arch" <arch at ovirt.org>
> > >>>>> Sent: Thursday, February 20, 2014 10:02:00 AM
> > >>>>> Subject: Re: ovirt-engine build failure
> > >>>>>
> > >>>>> Il 19/02/2014 16:51, Vojtech Szocs ha scritto:
> > >>>>>>
> > >>>>>>
> > >>>>>> ----- Original Message -----
> > >>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> > >>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> > >>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> > >>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> > >>>>>>> "arch" <arch at ovirt.org>
> > >>>>>>> Sent: Wednesday, February 19, 2014 2:43:44 PM
> > >>>>>>> Subject: Re: ovirt-engine build failure
> > >>>>>>>
> > >>>>>>> Il 19/02/2014 14:41, Vojtech Szocs ha scritto:
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> ----- Original Message -----
> > >>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> > >>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> > >>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> > >>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> > >>>>>>>>> "arch" <arch at ovirt.org>
> > >>>>>>>>> Sent: Wednesday, February 19, 2014 2:23:16 PM
> > >>>>>>>>> Subject: Re: ovirt-engine build failure
> > >>>>>>>>>
> > >>>>>>>>> Il 19/02/2014 14:19, Vojtech Szocs ha scritto:
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> ----- Original Message -----
> > >>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> > >>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
> > >>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> > >>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> > >>>>>>>>>>> "arch" <arch at ovirt.org>
> > >>>>>>>>>>> Sent: Wednesday, February 19, 2014 1:18:28 PM
> > >>>>>>>>>>> Subject: Re: ovirt-engine build failure
> > >>>>>>>>>>>
> > >>>>>>>>>>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto:
> > >>>>>>>>>>>> Hi, I checked GWT sources:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever
> > >>>>>>>>>>>>> computed
> > >>>>>>>>>>>>> for
> > >>>>>>>>>>>>> removeBrickCommitRequired
> > >>>>>>>>>>>>> [INFO] 	at
> > >>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> This happened because internal image cache for bundled
> > >>>>>>>>>>>> (composite
> > >>>>>>>>>>>> PNG)
> > >>>>>>>>>>>> images was corrput.
> > >>>>>>>>>>>> It seems to be caused by disk I/O error while reading file
> > >>>>>>>>>>>> images/glusterAsyncTasks/remove_brick_commit_required.png
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> [INFO] java.lang.NullPointerException
> > >>>>>>>>>>>>> [INFO] 	at
> > >>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> This happened because image data was null, again possible disk
> > >>>>>>>>>>>> I/O
> > >>>>>>>>>>>> error
> > >>>>>>>>>>>> while reading image file.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Can we enable "TRACE" logging during GWT compilation?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>   -Dgwt.logLevel=TRACE
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Building master right now with your suggestions here:
> > >>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/
> > >>>>>>>>>>
> > >>>>>>>>>> Looking at
> > >>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host/console
> > >>>>>>>>>>
> > >>>>>>>>>> So far I see no errors.. There should be total 5 browsers x 8
> > >>>>>>>>>> locales
> > >>>>>>>>>> =
> > >>>>>>>>>> 40
> > >>>>>>>>>> GWT permutations.
> > >>>>>>>>>
> > >>>>>>>>> So no changes with and without
> > >>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera.
> > >>>>>>>>> We've still 40 permutations. Maybe you defined
> > >>>>>>>>> userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the POM
> > >>>>>>>>> files?
> > >>>>>>>>
> > >>>>>>>> You are right. I checked frontend/webadmin/modules/pom.xml -> the
> > >>>>>>>> "gwt.userAgent"
> > >>>>>>>> property is copied into "locale" GWT property in
> > >>>>>>>> {WebAdmin,UserPortal}.gwt.xml
> > >>>>>>>
> > >>>>>>> Ok, so I'll remove -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera
> > >>>>>>> from
> > >>>>>>> build
> > >>>>>>> command.
> > >>>>>>
> > >>>>>> Agreed.
> > >>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> This:
> > >>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/console
> > >>>>>>>>> failed, you can look at it, it has -Dgwt.logLevel=TRACE -X.
> > >>>>>>>>
> > >>>>>>>> Hm, it contains:
> > >>>>>>>>
> > >>>>>>>> [DEBUG]   (f) logLevel = INFO
> > >>>>>>>>
> > >>>>>>>> Somehow the TRACE log level for GWT wasn't applied.. not sure why.
> > >>>>>>>> Let
> > >>>>>>>> me
> > >>>>>>>> check
> > >>>>>>>> this on local build.
> > >>>>>>>
> > >>>>>>> Ok, let me know if I've to change command line.
> > >>>>>>
> > >>>>>> I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/
> > >>>>>>
> > >>>>>> When it's merged, setting "gwt.logLevel" will work (as you did
> > >>>>>> before)
> > >>>>>> and we'll be able to see more details in the build log.
> > >>>>>
> > >>>>> Here is the new build with your patch merged:
> > >>>>>
> > >>>>> http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/consoleFull
> > >>>>
> > >>>> Thanks. I did some more investigation on this.
> > >>>>
> > >>>> Again it's NullPointerException when rebinding ApplicationResources,
> > >>>> but
> > >>>> this time for different image file: ApplicationResources.wrenchImage
> > >>>> ("images/wrench.png").
> > >>>>
> > >>>> Looking at stacktrace & GWT sources, the problematic code seems to be
> > >>>> (AbstractResourceContext:74):
> > >>>>
> > >>>>   byte[] bytes = Util.readURLAsBytes(resource);
> > >>>>
> > >>>> where "resource" is (ImageResourceGenerator:311):
> > >>>>
> > >>>>   localized.getUrl(); // LocalizedImage localized
> > >>>>
> > >>>> Note: in oVirt UI we don't use localized images (i.e.
> > >>>> "wrench_fr_FR.png")
> > >>>> so "localized" in this case points to "wrench.png".
> > >>>>
> > >>>> So the problem seems to be reading this file:
> > >>>>
> > >>>>   frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png
> > >>>>
> > >>>> Since we're having many GWT permutations, this file ^^ (along with
> > >>>> similar
> > >>>> ones) is attempted to be read multiple times *from the same (single)
> > >>>> GWT
> > >>>> compiler Java process, before actual permutation worker Java
> > >>>> sub-processes
> > >>>> are spawned*. (More precisely, above NullPointerException happens at
> > >>>> deferred binding phase which precedes actual GWT compilation per each
> > >>>> permutation.)
> > >>>>
> > >>>> I think the root cause is Util.readURLAsBytes method used internally
> > >>>> by
> > >>>> GWT
> > >>>> compiler:
> > >>>>
> > >>>>   public static byte[] readURLAsBytes(URL url) {
> > >>>>     try {
> > >>>>       URLConnection conn = url.openConnection();
> > >>>>       conn.setUseCaches(false);
> > >>>>       return readURLConnectionAsBytes(conn);
> > >>>>     } catch (IOException e) {
> > >>>>       return null; // This causes NullPointerException, but this
> > >>>>       exception
> > >>>>       isn't logged (wtf, GWT!)
> > >>>>     }
> > >>>>   }
> > >>>>
> > >>>> To summarize, I'll need to write another patch which replaces
> > >>>> Util.readURLAsBytes implementation via AOP and prints IOException
> > >>>> details.
> > >>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>>
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> #
> > >>>>>>>>>>> CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml
> > >>>>>>>>>>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M
> > >>>>>>>>>>> -XX:MaxPermSize=1024m
> > >>>>>>>>>>> -Dgwt.compiler.localWorkers=1"
> > >>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}"
> > >>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D
> > >>>>>>>>>>> "ovirt_build_extra_flags
> > >>>>>>>>>>> -gs
> > >>>>>>>>>>> ${CI_MAVEN_SETTINGS}
> > >>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera
> > >>>>>>>>>>> -Dgwt.logLevel=TRACE -X" --rebuild rpmbuild/SRPMS/*.src.rpm
> > >>>>>>>>>>>
> > >>>>>>>>>>> Let me know if you need anything else.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Vojtech
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> ----- Original Message -----
> > >>>>>>>>>>>>> From: "Vojtech Szocs" <vszocs at redhat.com>
> > >>>>>>>>>>>>> To: "Sandro Bonazzola" <sbonazzo at redhat.com>
> > >>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
> > >>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
> > >>>>>>>>>>>>> "arch" <arch at ovirt.org>
> > >>>>>>>>>>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM
> > >>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> In your log I see following error:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> ...
> > >>>>>>>>>>>>> [INFO]    Computing all possible rebind results for
> > >>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> > >>>>>>>>>>>>> [INFO]       Rebinding
> > >>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
> > >>>>>>>>>>>>> [INFO]          Invoking generator
> > >>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
> > >>>>>>>>>>>>> [INFO]             [ERROR] Generator
> > >>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
> > >>>>>>>>>>>>> threw
> > >>>>>>>>>>>>> an exception while rebinding
> > >>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> > >>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever
> > >>>>>>>>>>>>> computed
> > >>>>>>>>>>>>> for
> > >>>>>>>>>>>>> removeBrickCommitRequired
> > >>>>>>>>>>>>> [INFO] 	at
> > >>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
> > >>>>>>>>>>>>> ...
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I checked again logs at
> > >>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host/consoleFull
> > >>>>>>>>>>>>> and indeed there is a similar error:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> ...
> > >>>>>>>>>>>>> [INFO]    Computing all possible rebind results for
> > >>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> > >>>>>>>>>>>>> [INFO]       Rebinding
> > >>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
> > >>>>>>>>>>>>> [INFO]          Invoking generator
> > >>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
> > >>>>>>>>>>>>> [INFO]             [ERROR] Generator
> > >>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
> > >>>>>>>>>>>>> threw
> > >>>>>>>>>>>>> an exception while rebinding
> > >>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
> > >>>>>>>>>>>>> [INFO] java.lang.NullPointerException
> > >>>>>>>>>>>>> [INFO] 	at
> > >>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
> > >>>>>>>>>>>>> ...
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> NullPointerException when processing ApplicationResources but
> > >>>>>>>>>>>>> occuring
> > >>>>>>>>>>>>> from
> > >>>>>>>>>>>>> different places (ImageResourceGenerator vs.
> > >>>>>>>>>>>>> InlineResourceContext)
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Now I understand why Maven build fails without any details:
> > >>>>>>>>>>>>> - GWT deferred binding fails for ApplicationResources (i.e.
> > >>>>>>>>>>>>> uncaught
> > >>>>>>>>>>>>> NPE)
> > >>>>>>>>>>>>> - GWT compilation proceeds, fails to link with
> > >>>>>>>>>>>>> ApplicationResources
> > >>>>>>>>>>>>> - GWT compilation task fails with non-OK status code
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> So the issue here is GWT deferred binding generator
> > >>>>>>>>>>>>> (InlineClientBundleGenerator) failure during GWT compilation.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> GWT compiler writes extra Java files during deferred binding
> > >>>>>>>>>>>>> phase, so this could point to issue with local environment..
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I will try to investigate this and look at GWT source code.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> --
> > >>>>>>>>>>> Sandro Bonazzola
> > >>>>>>>>>>> Better technology. Faster innovation. Powered by community
> > >>>>>>>>>>> collaboration.
> > >>>>>>>>>>> See how it works at redhat.com
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> --
> > >>>>>>>>> Sandro Bonazzola
> > >>>>>>>>> Better technology. Faster innovation. Powered by community
> > >>>>>>>>> collaboration.
> > >>>>>>>>> See how it works at redhat.com
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> --
> > >>>>>>> Sandro Bonazzola
> > >>>>>>> Better technology. Faster innovation. Powered by community
> > >>>>>>> collaboration.
> > >>>>>>> See how it works at redhat.com
> > >>>>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> Sandro Bonazzola
> > >>>>> Better technology. Faster innovation. Powered by community
> > >>>>> collaboration.
> > >>>>> See how it works at redhat.com
> > >>>>>
> > >>>
> > >>>
> > >>> --
> > >>> Sandro Bonazzola
> > >>> Better technology. Faster innovation. Powered by community
> > >>> collaboration.
> > >>> See how it works at redhat.com
> > >>>
> > > 
> > > 
> > 
> > 
> > --
> > Sandro Bonazzola
> > Better technology. Faster innovation. Powered by community collaboration.
> > See how it works at redhat.com
> > 
> _______________________________________________
> Arch mailing list
> Arch at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/arch
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-TEST-v2.patch
Type: text/x-patch
Size: 5349 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/arch/attachments/20140225/b187dd24/attachment.bin>

More information about the Arch mailing list