[urgent] Re: ovirt-engine build failure

Sandro Bonazzola sbonazzo at redhat.com
Mon Feb 24 08:23:51 UTC 2014 

Il 24/02/2014 09:08, Sandro Bonazzola ha scritto:
> Top posting,
> 
> Please give this task really urgent priority, as you can see:
> 
> http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/
> and
> http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/
> 
> are failing consistently so we don't have nightly builds.
> 
> 

CCing infra.

> 
> 
> Il 20/02/2014 13:34, Vojtech Szocs ha scritto:
>>
>>
>> ----- Original Message -----
>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez" <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>> "arch" <arch at ovirt.org>
>>> Sent: Thursday, February 20, 2014 10:02:00 AM
>>> Subject: Re: ovirt-engine build failure
>>>
>>> Il 19/02/2014 16:51, Vojtech Szocs ha scritto:
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>> "arch" <arch at ovirt.org>
>>>>> Sent: Wednesday, February 19, 2014 2:43:44 PM
>>>>> Subject: Re: ovirt-engine build failure
>>>>>
>>>>> Il 19/02/2014 14:41, Vojtech Szocs ha scritto:
>>>>>>
>>>>>>
>>>>>> ----- Original Message -----
>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>>>> "arch" <arch at ovirt.org>
>>>>>>> Sent: Wednesday, February 19, 2014 2:23:16 PM
>>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>>
>>>>>>> Il 19/02/2014 14:19, Vojtech Szocs ha scritto:
>>>>>>>>
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>>>>>> "arch" <arch at ovirt.org>
>>>>>>>>> Sent: Wednesday, February 19, 2014 1:18:28 PM
>>>>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>>>>
>>>>>>>>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto:
>>>>>>>>>> Hi, I checked GWT sources:
>>>>>>>>>>
>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed
>>>>>>>>>>> for
>>>>>>>>>>> removeBrickCommitRequired
>>>>>>>>>>> [INFO] 	at
>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
>>>>>>>>>>
>>>>>>>>>> This happened because internal image cache for bundled (composite
>>>>>>>>>> PNG)
>>>>>>>>>> images was corrput.
>>>>>>>>>> It seems to be caused by disk I/O error while reading file
>>>>>>>>>> images/glusterAsyncTasks/remove_brick_commit_required.png
>>>>>>>>>>
>>>>>>>>>>> [INFO] java.lang.NullPointerException
>>>>>>>>>>> [INFO] 	at
>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
>>>>>>>>>>
>>>>>>>>>> This happened because image data was null, again possible disk I/O
>>>>>>>>>> error
>>>>>>>>>> while reading image file.
>>>>>>>>>>
>>>>>>>>>> Can we enable "TRACE" logging during GWT compilation?
>>>>>>>>>>
>>>>>>>>>>   -Dgwt.logLevel=TRACE
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Building master right now with your suggestions here:
>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/
>>>>>>>>
>>>>>>>> Looking at
>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host/console
>>>>>>>>
>>>>>>>> So far I see no errors.. There should be total 5 browsers x 8 locales =
>>>>>>>> 40
>>>>>>>> GWT permutations.
>>>>>>>
>>>>>>> So no changes with and without
>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera.
>>>>>>> We've still 40 permutations. Maybe you defined
>>>>>>> userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the POM files?
>>>>>>
>>>>>> You are right. I checked frontend/webadmin/modules/pom.xml -> the
>>>>>> "gwt.userAgent"
>>>>>> property is copied into "locale" GWT property in
>>>>>> {WebAdmin,UserPortal}.gwt.xml
>>>>>
>>>>> Ok, so I'll remove -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera from
>>>>> build
>>>>> command.
>>>>
>>>> Agreed.
>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>> This:
>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/console
>>>>>>> failed, you can look at it, it has -Dgwt.logLevel=TRACE -X.
>>>>>>
>>>>>> Hm, it contains:
>>>>>>
>>>>>> [DEBUG]   (f) logLevel = INFO
>>>>>>
>>>>>> Somehow the TRACE log level for GWT wasn't applied.. not sure why. Let me
>>>>>> check
>>>>>> this on local build.
>>>>>
>>>>> Ok, let me know if I've to change command line.
>>>>
>>>> I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/
>>>>
>>>> When it's merged, setting "gwt.logLevel" will work (as you did before)
>>>> and we'll be able to see more details in the build log.
>>>
>>> Here is the new build with your patch merged:
>>>
>>> http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/consoleFull
>>
>> Thanks. I did some more investigation on this.
>>
>> Again it's NullPointerException when rebinding ApplicationResources, but this time for different image file: ApplicationResources.wrenchImage ("images/wrench.png").
>>
>> Looking at stacktrace & GWT sources, the problematic code seems to be (AbstractResourceContext:74):
>>
>>   byte[] bytes = Util.readURLAsBytes(resource);
>>
>> where "resource" is (ImageResourceGenerator:311):
>>
>>   localized.getUrl(); // LocalizedImage localized
>>
>> Note: in oVirt UI we don't use localized images (i.e. "wrench_fr_FR.png") so "localized" in this case points to "wrench.png".
>>
>> So the problem seems to be reading this file:
>>
>>   frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png
>>
>> Since we're having many GWT permutations, this file ^^ (along with similar ones) is attempted to be read multiple times *from the same (single) GWT compiler Java process, before actual permutation worker Java sub-processes are spawned*. (More precisely, above NullPointerException happens at deferred binding phase which precedes actual GWT compilation per each permutation.)
>>
>> I think the root cause is Util.readURLAsBytes method used internally by GWT compiler:
>>
>>   public static byte[] readURLAsBytes(URL url) {
>>     try {
>>       URLConnection conn = url.openConnection();
>>       conn.setUseCaches(false);
>>       return readURLConnectionAsBytes(conn);
>>     } catch (IOException e) {
>>       return null; // This causes NullPointerException, but this exception isn't logged (wtf, GWT!)
>>     }
>>   }
>>
>> To summarize, I'll need to write another patch which replaces Util.readURLAsBytes implementation via AOP and prints IOException details.
>>
>>>
>>>
>>>
>>>
>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> # CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml
>>>>>>>>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M -XX:MaxPermSize=1024m
>>>>>>>>> -Dgwt.compiler.localWorkers=1"
>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}"
>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D "ovirt_build_extra_flags
>>>>>>>>> -gs
>>>>>>>>> ${CI_MAVEN_SETTINGS} -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera
>>>>>>>>> -Dgwt.logLevel=TRACE -X" --rebuild rpmbuild/SRPMS/*.src.rpm
>>>>>>>>>
>>>>>>>>> Let me know if you need anything else.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Vojtech
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>> From: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>>>>> To: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>>>>>>>> "arch" <arch at ovirt.org>
>>>>>>>>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM
>>>>>>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>>>>>>
>>>>>>>>>>> In your log I see following error:
>>>>>>>>>>>
>>>>>>>>>>> ...
>>>>>>>>>>> [INFO]    Computing all possible rebind results for
>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
>>>>>>>>>>> [INFO]       Rebinding
>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
>>>>>>>>>>> [INFO]          Invoking generator
>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
>>>>>>>>>>> [INFO]             [ERROR] Generator
>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
>>>>>>>>>>> threw
>>>>>>>>>>> an exception while rebinding
>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed
>>>>>>>>>>> for
>>>>>>>>>>> removeBrickCommitRequired
>>>>>>>>>>> [INFO] 	at
>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
>>>>>>>>>>> ...
>>>>>>>>>>>
>>>>>>>>>>> I checked again logs at
>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host/consoleFull
>>>>>>>>>>> and indeed there is a similar error:
>>>>>>>>>>>
>>>>>>>>>>> ...
>>>>>>>>>>> [INFO]    Computing all possible rebind results for
>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
>>>>>>>>>>> [INFO]       Rebinding
>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
>>>>>>>>>>> [INFO]          Invoking generator
>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
>>>>>>>>>>> [INFO]             [ERROR] Generator
>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
>>>>>>>>>>> threw
>>>>>>>>>>> an exception while rebinding
>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
>>>>>>>>>>> [INFO] java.lang.NullPointerException
>>>>>>>>>>> [INFO] 	at
>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
>>>>>>>>>>> ...
>>>>>>>>>>>
>>>>>>>>>>> NullPointerException when processing ApplicationResources but
>>>>>>>>>>> occuring
>>>>>>>>>>> from
>>>>>>>>>>> different places (ImageResourceGenerator vs. InlineResourceContext)
>>>>>>>>>>>
>>>>>>>>>>> Now I understand why Maven build fails without any details:
>>>>>>>>>>> - GWT deferred binding fails for ApplicationResources (i.e. uncaught
>>>>>>>>>>> NPE)
>>>>>>>>>>> - GWT compilation proceeds, fails to link with ApplicationResources
>>>>>>>>>>> - GWT compilation task fails with non-OK status code
>>>>>>>>>>>
>>>>>>>>>>> So the issue here is GWT deferred binding generator
>>>>>>>>>>> (InlineClientBundleGenerator) failure during GWT compilation.
>>>>>>>>>>>
>>>>>>>>>>> GWT compiler writes extra Java files during deferred binding
>>>>>>>>>>> phase, so this could point to issue with local environment..
>>>>>>>>>>>
>>>>>>>>>>> I will try to investigate this and look at GWT source code.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Sandro Bonazzola
>>>>>>>>> Better technology. Faster innovation. Powered by community
>>>>>>>>> collaboration.
>>>>>>>>> See how it works at redhat.com
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Sandro Bonazzola
>>>>>>> Better technology. Faster innovation. Powered by community
>>>>>>> collaboration.
>>>>>>> See how it works at redhat.com
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sandro Bonazzola
>>>>> Better technology. Faster innovation. Powered by community collaboration.
>>>>> See how it works at redhat.com
>>>>>
>>>
>>>
>>> --
>>> Sandro Bonazzola
>>> Better technology. Faster innovation. Powered by community collaboration.
>>> See how it works at redhat.com
>>>
> 
> 


-- 
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com

More information about the Infra mailing list