[urgent] Re: ovirt-engine build failure

Sandro Bonazzola sbonazzo at redhat.com
Tue Feb 25 08:02:11 UTC 2014 

Il 25/02/2014 08:47, Sandro Bonazzola ha scritto:
> Il 24/02/2014 18:28, Vojtech Szocs ha scritto:
>>
>>
>> ----- Original Message -----
>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez" <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>> "arch" <arch at ovirt.org>
>>> Sent: Monday, February 24, 2014 9:08:09 AM
>>> Subject: [urgent] Re: ovirt-engine build failure
>>>
>>> Top posting,
>>>
>>> Please give this task really urgent priority, as you can see:
>>>
>>> http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/
>>> and
>>> http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/
>>>
>>> are failing consistently so we don't have nightly builds.
>>
>> (CC'ing Einav)
>>
>> Attached TEST patch, please apply & rebuild ovirt-engine as usual.
>>
>> This patch should cause following output during GWT compilation:
>>
>> [ERROR] ######### Util.readURLAsBytes file:/home/vszocs/work/ovirt-engine/frontend/webadmin/modules/webadmin/target/generated-sources/gwt/org/ovirt/engine/ui/webadmin/images/search_button.png
>>
>> If IOException occurs, exception stacktrace will be printed right below that line.
>>
>> Note: please ignore the [ERROR] prefix, it's a technical issue, more details at https://www.mail-archive.com/engine-devel@ovirt.org/msg05735.html
>>
>> I just did a local upstream build with TEST patch applied, GWT compilation passed OK.
> 
> build job running with your patch here: http://jenkins.ovirt.org/job/manual-build-tarball/235/
> waiting for a free executor right now.
> Let's see how it goes.

Failed on CentOS: http://jenkins.ovirt.org/job/manual-build-tarball/235/label=centos6-host/consoleFull






> 
> 
> 
>>
>>>
>>>
>>>
>>>
>>> Il 20/02/2014 13:34, Vojtech Szocs ha scritto:
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>> "arch" <arch at ovirt.org>
>>>>> Sent: Thursday, February 20, 2014 10:02:00 AM
>>>>> Subject: Re: ovirt-engine build failure
>>>>>
>>>>> Il 19/02/2014 16:51, Vojtech Szocs ha scritto:
>>>>>>
>>>>>>
>>>>>> ----- Original Message -----
>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>>>> "arch" <arch at ovirt.org>
>>>>>>> Sent: Wednesday, February 19, 2014 2:43:44 PM
>>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>>
>>>>>>> Il 19/02/2014 14:41, Vojtech Szocs ha scritto:
>>>>>>>>
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>>>>>> "arch" <arch at ovirt.org>
>>>>>>>>> Sent: Wednesday, February 19, 2014 2:23:16 PM
>>>>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>>>>
>>>>>>>>> Il 19/02/2014 14:19, Vojtech Szocs ha scritto:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>>>>>>>> "arch" <arch at ovirt.org>
>>>>>>>>>>> Sent: Wednesday, February 19, 2014 1:18:28 PM
>>>>>>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>>>>>>
>>>>>>>>>>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto:
>>>>>>>>>>>> Hi, I checked GWT sources:
>>>>>>>>>>>>
>>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed
>>>>>>>>>>>>> for
>>>>>>>>>>>>> removeBrickCommitRequired
>>>>>>>>>>>>> [INFO] 	at
>>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
>>>>>>>>>>>>
>>>>>>>>>>>> This happened because internal image cache for bundled (composite
>>>>>>>>>>>> PNG)
>>>>>>>>>>>> images was corrput.
>>>>>>>>>>>> It seems to be caused by disk I/O error while reading file
>>>>>>>>>>>> images/glusterAsyncTasks/remove_brick_commit_required.png
>>>>>>>>>>>>
>>>>>>>>>>>>> [INFO] java.lang.NullPointerException
>>>>>>>>>>>>> [INFO] 	at
>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
>>>>>>>>>>>>
>>>>>>>>>>>> This happened because image data was null, again possible disk I/O
>>>>>>>>>>>> error
>>>>>>>>>>>> while reading image file.
>>>>>>>>>>>>
>>>>>>>>>>>> Can we enable "TRACE" logging during GWT compilation?
>>>>>>>>>>>>
>>>>>>>>>>>>   -Dgwt.logLevel=TRACE
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Building master right now with your suggestions here:
>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/
>>>>>>>>>>
>>>>>>>>>> Looking at
>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host/console
>>>>>>>>>>
>>>>>>>>>> So far I see no errors.. There should be total 5 browsers x 8 locales
>>>>>>>>>> =
>>>>>>>>>> 40
>>>>>>>>>> GWT permutations.
>>>>>>>>>
>>>>>>>>> So no changes with and without
>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera.
>>>>>>>>> We've still 40 permutations. Maybe you defined
>>>>>>>>> userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the POM files?
>>>>>>>>
>>>>>>>> You are right. I checked frontend/webadmin/modules/pom.xml -> the
>>>>>>>> "gwt.userAgent"
>>>>>>>> property is copied into "locale" GWT property in
>>>>>>>> {WebAdmin,UserPortal}.gwt.xml
>>>>>>>
>>>>>>> Ok, so I'll remove -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera from
>>>>>>> build
>>>>>>> command.
>>>>>>
>>>>>> Agreed.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> This:
>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/console
>>>>>>>>> failed, you can look at it, it has -Dgwt.logLevel=TRACE -X.
>>>>>>>>
>>>>>>>> Hm, it contains:
>>>>>>>>
>>>>>>>> [DEBUG]   (f) logLevel = INFO
>>>>>>>>
>>>>>>>> Somehow the TRACE log level for GWT wasn't applied.. not sure why. Let
>>>>>>>> me
>>>>>>>> check
>>>>>>>> this on local build.
>>>>>>>
>>>>>>> Ok, let me know if I've to change command line.
>>>>>>
>>>>>> I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/
>>>>>>
>>>>>> When it's merged, setting "gwt.logLevel" will work (as you did before)
>>>>>> and we'll be able to see more details in the build log.
>>>>>
>>>>> Here is the new build with your patch merged:
>>>>>
>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/consoleFull
>>>>
>>>> Thanks. I did some more investigation on this.
>>>>
>>>> Again it's NullPointerException when rebinding ApplicationResources, but
>>>> this time for different image file: ApplicationResources.wrenchImage
>>>> ("images/wrench.png").
>>>>
>>>> Looking at stacktrace & GWT sources, the problematic code seems to be
>>>> (AbstractResourceContext:74):
>>>>
>>>>   byte[] bytes = Util.readURLAsBytes(resource);
>>>>
>>>> where "resource" is (ImageResourceGenerator:311):
>>>>
>>>>   localized.getUrl(); // LocalizedImage localized
>>>>
>>>> Note: in oVirt UI we don't use localized images (i.e. "wrench_fr_FR.png")
>>>> so "localized" in this case points to "wrench.png".
>>>>
>>>> So the problem seems to be reading this file:
>>>>
>>>>   frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png
>>>>
>>>> Since we're having many GWT permutations, this file ^^ (along with similar
>>>> ones) is attempted to be read multiple times *from the same (single) GWT
>>>> compiler Java process, before actual permutation worker Java sub-processes
>>>> are spawned*. (More precisely, above NullPointerException happens at
>>>> deferred binding phase which precedes actual GWT compilation per each
>>>> permutation.)
>>>>
>>>> I think the root cause is Util.readURLAsBytes method used internally by GWT
>>>> compiler:
>>>>
>>>>   public static byte[] readURLAsBytes(URL url) {
>>>>     try {
>>>>       URLConnection conn = url.openConnection();
>>>>       conn.setUseCaches(false);
>>>>       return readURLConnectionAsBytes(conn);
>>>>     } catch (IOException e) {
>>>>       return null; // This causes NullPointerException, but this exception
>>>>       isn't logged (wtf, GWT!)
>>>>     }
>>>>   }
>>>>
>>>> To summarize, I'll need to write another patch which replaces
>>>> Util.readURLAsBytes implementation via AOP and prints IOException details.
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> # CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml
>>>>>>>>>>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M -XX:MaxPermSize=1024m
>>>>>>>>>>> -Dgwt.compiler.localWorkers=1"
>>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}"
>>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D "ovirt_build_extra_flags
>>>>>>>>>>> -gs
>>>>>>>>>>> ${CI_MAVEN_SETTINGS} -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera
>>>>>>>>>>> -Dgwt.logLevel=TRACE -X" --rebuild rpmbuild/SRPMS/*.src.rpm
>>>>>>>>>>>
>>>>>>>>>>> Let me know if you need anything else.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Vojtech
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>> From: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>>>>>>> To: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>>>>>>>>>> "arch" <arch at ovirt.org>
>>>>>>>>>>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM
>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>>>>>>>>
>>>>>>>>>>>>> In your log I see following error:
>>>>>>>>>>>>>
>>>>>>>>>>>>> ...
>>>>>>>>>>>>> [INFO]    Computing all possible rebind results for
>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
>>>>>>>>>>>>> [INFO]       Rebinding
>>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
>>>>>>>>>>>>> [INFO]          Invoking generator
>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
>>>>>>>>>>>>> [INFO]             [ERROR] Generator
>>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
>>>>>>>>>>>>> threw
>>>>>>>>>>>>> an exception while rebinding
>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
>>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed
>>>>>>>>>>>>> for
>>>>>>>>>>>>> removeBrickCommitRequired
>>>>>>>>>>>>> [INFO] 	at
>>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
>>>>>>>>>>>>> ...
>>>>>>>>>>>>>
>>>>>>>>>>>>> I checked again logs at
>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host/consoleFull
>>>>>>>>>>>>> and indeed there is a similar error:
>>>>>>>>>>>>>
>>>>>>>>>>>>> ...
>>>>>>>>>>>>> [INFO]    Computing all possible rebind results for
>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
>>>>>>>>>>>>> [INFO]       Rebinding
>>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
>>>>>>>>>>>>> [INFO]          Invoking generator
>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
>>>>>>>>>>>>> [INFO]             [ERROR] Generator
>>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
>>>>>>>>>>>>> threw
>>>>>>>>>>>>> an exception while rebinding
>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
>>>>>>>>>>>>> [INFO] java.lang.NullPointerException
>>>>>>>>>>>>> [INFO] 	at
>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
>>>>>>>>>>>>> ...
>>>>>>>>>>>>>
>>>>>>>>>>>>> NullPointerException when processing ApplicationResources but
>>>>>>>>>>>>> occuring
>>>>>>>>>>>>> from
>>>>>>>>>>>>> different places (ImageResourceGenerator vs.
>>>>>>>>>>>>> InlineResourceContext)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Now I understand why Maven build fails without any details:
>>>>>>>>>>>>> - GWT deferred binding fails for ApplicationResources (i.e.
>>>>>>>>>>>>> uncaught
>>>>>>>>>>>>> NPE)
>>>>>>>>>>>>> - GWT compilation proceeds, fails to link with
>>>>>>>>>>>>> ApplicationResources
>>>>>>>>>>>>> - GWT compilation task fails with non-OK status code
>>>>>>>>>>>>>
>>>>>>>>>>>>> So the issue here is GWT deferred binding generator
>>>>>>>>>>>>> (InlineClientBundleGenerator) failure during GWT compilation.
>>>>>>>>>>>>>
>>>>>>>>>>>>> GWT compiler writes extra Java files during deferred binding
>>>>>>>>>>>>> phase, so this could point to issue with local environment..
>>>>>>>>>>>>>
>>>>>>>>>>>>> I will try to investigate this and look at GWT source code.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Sandro Bonazzola
>>>>>>>>>>> Better technology. Faster innovation. Powered by community
>>>>>>>>>>> collaboration.
>>>>>>>>>>> See how it works at redhat.com
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Sandro Bonazzola
>>>>>>>>> Better technology. Faster innovation. Powered by community
>>>>>>>>> collaboration.
>>>>>>>>> See how it works at redhat.com
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Sandro Bonazzola
>>>>>>> Better technology. Faster innovation. Powered by community
>>>>>>> collaboration.
>>>>>>> See how it works at redhat.com
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sandro Bonazzola
>>>>> Better technology. Faster innovation. Powered by community collaboration.
>>>>> See how it works at redhat.com
>>>>>
>>>
>>>
>>> --
>>> Sandro Bonazzola
>>> Better technology. Faster innovation. Powered by community collaboration.
>>> See how it works at redhat.com
>>>
> 
> 


-- 
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com

More information about the Arch mailing list