[urgent] Re: ovirt-engine build failure

Sandro Bonazzola sbonazzo at redhat.com
Tue Feb 25 07:47:53 UTC 2014 

Il 24/02/2014 18:28, Vojtech Szocs ha scritto:
> 
> 
> ----- Original Message -----
>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>> To: "Vojtech Szocs" <vszocs at redhat.com>
>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez" <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>> "arch" <arch at ovirt.org>
>> Sent: Monday, February 24, 2014 9:08:09 AM
>> Subject: [urgent] Re: ovirt-engine build failure
>>
>> Top posting,
>>
>> Please give this task really urgent priority, as you can see:
>>
>> http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/
>> and
>> http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/
>>
>> are failing consistently so we don't have nightly builds.
> 
> (CC'ing Einav)
> 
> Attached TEST patch, please apply & rebuild ovirt-engine as usual.
> 
> This patch should cause following output during GWT compilation:
> 
> [ERROR] ######### Util.readURLAsBytes file:/home/vszocs/work/ovirt-engine/frontend/webadmin/modules/webadmin/target/generated-sources/gwt/org/ovirt/engine/ui/webadmin/images/search_button.png
> 
> If IOException occurs, exception stacktrace will be printed right below that line.
> 
> Note: please ignore the [ERROR] prefix, it's a technical issue, more details at https://www.mail-archive.com/engine-devel@ovirt.org/msg05735.html
> 
> I just did a local upstream build with TEST patch applied, GWT compilation passed OK.

build job running with your patch here: http://jenkins.ovirt.org/job/manual-build-tarball/235/
waiting for a free executor right now.
Let's see how it goes.



> 
>>
>>
>>
>>
>> Il 20/02/2014 13:34, Vojtech Szocs ha scritto:
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>> "arch" <arch at ovirt.org>
>>>> Sent: Thursday, February 20, 2014 10:02:00 AM
>>>> Subject: Re: ovirt-engine build failure
>>>>
>>>> Il 19/02/2014 16:51, Vojtech Szocs ha scritto:
>>>>>
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>>> "arch" <arch at ovirt.org>
>>>>>> Sent: Wednesday, February 19, 2014 2:43:44 PM
>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>
>>>>>> Il 19/02/2014 14:41, Vojtech Szocs ha scritto:
>>>>>>>
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>>>>> "arch" <arch at ovirt.org>
>>>>>>>> Sent: Wednesday, February 19, 2014 2:23:16 PM
>>>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>>>
>>>>>>>> Il 19/02/2014 14:19, Vojtech Szocs ha scritto:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----- Original Message -----
>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>>>>>>> "arch" <arch at ovirt.org>
>>>>>>>>>> Sent: Wednesday, February 19, 2014 1:18:28 PM
>>>>>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>>>>>
>>>>>>>>>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto:
>>>>>>>>>>> Hi, I checked GWT sources:
>>>>>>>>>>>
>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed
>>>>>>>>>>>> for
>>>>>>>>>>>> removeBrickCommitRequired
>>>>>>>>>>>> [INFO] 	at
>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
>>>>>>>>>>>
>>>>>>>>>>> This happened because internal image cache for bundled (composite
>>>>>>>>>>> PNG)
>>>>>>>>>>> images was corrput.
>>>>>>>>>>> It seems to be caused by disk I/O error while reading file
>>>>>>>>>>> images/glusterAsyncTasks/remove_brick_commit_required.png
>>>>>>>>>>>
>>>>>>>>>>>> [INFO] java.lang.NullPointerException
>>>>>>>>>>>> [INFO] 	at
>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
>>>>>>>>>>>
>>>>>>>>>>> This happened because image data was null, again possible disk I/O
>>>>>>>>>>> error
>>>>>>>>>>> while reading image file.
>>>>>>>>>>>
>>>>>>>>>>> Can we enable "TRACE" logging during GWT compilation?
>>>>>>>>>>>
>>>>>>>>>>>   -Dgwt.logLevel=TRACE
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Building master right now with your suggestions here:
>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/
>>>>>>>>>
>>>>>>>>> Looking at
>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host/console
>>>>>>>>>
>>>>>>>>> So far I see no errors.. There should be total 5 browsers x 8 locales
>>>>>>>>> =
>>>>>>>>> 40
>>>>>>>>> GWT permutations.
>>>>>>>>
>>>>>>>> So no changes with and without
>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera.
>>>>>>>> We've still 40 permutations. Maybe you defined
>>>>>>>> userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the POM files?
>>>>>>>
>>>>>>> You are right. I checked frontend/webadmin/modules/pom.xml -> the
>>>>>>> "gwt.userAgent"
>>>>>>> property is copied into "locale" GWT property in
>>>>>>> {WebAdmin,UserPortal}.gwt.xml
>>>>>>
>>>>>> Ok, so I'll remove -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera from
>>>>>> build
>>>>>> command.
>>>>>
>>>>> Agreed.
>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> This:
>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/console
>>>>>>>> failed, you can look at it, it has -Dgwt.logLevel=TRACE -X.
>>>>>>>
>>>>>>> Hm, it contains:
>>>>>>>
>>>>>>> [DEBUG]   (f) logLevel = INFO
>>>>>>>
>>>>>>> Somehow the TRACE log level for GWT wasn't applied.. not sure why. Let
>>>>>>> me
>>>>>>> check
>>>>>>> this on local build.
>>>>>>
>>>>>> Ok, let me know if I've to change command line.
>>>>>
>>>>> I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/
>>>>>
>>>>> When it's merged, setting "gwt.logLevel" will work (as you did before)
>>>>> and we'll be able to see more details in the build log.
>>>>
>>>> Here is the new build with your patch merged:
>>>>
>>>> http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/consoleFull
>>>
>>> Thanks. I did some more investigation on this.
>>>
>>> Again it's NullPointerException when rebinding ApplicationResources, but
>>> this time for different image file: ApplicationResources.wrenchImage
>>> ("images/wrench.png").
>>>
>>> Looking at stacktrace & GWT sources, the problematic code seems to be
>>> (AbstractResourceContext:74):
>>>
>>>   byte[] bytes = Util.readURLAsBytes(resource);
>>>
>>> where "resource" is (ImageResourceGenerator:311):
>>>
>>>   localized.getUrl(); // LocalizedImage localized
>>>
>>> Note: in oVirt UI we don't use localized images (i.e. "wrench_fr_FR.png")
>>> so "localized" in this case points to "wrench.png".
>>>
>>> So the problem seems to be reading this file:
>>>
>>>   frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png
>>>
>>> Since we're having many GWT permutations, this file ^^ (along with similar
>>> ones) is attempted to be read multiple times *from the same (single) GWT
>>> compiler Java process, before actual permutation worker Java sub-processes
>>> are spawned*. (More precisely, above NullPointerException happens at
>>> deferred binding phase which precedes actual GWT compilation per each
>>> permutation.)
>>>
>>> I think the root cause is Util.readURLAsBytes method used internally by GWT
>>> compiler:
>>>
>>>   public static byte[] readURLAsBytes(URL url) {
>>>     try {
>>>       URLConnection conn = url.openConnection();
>>>       conn.setUseCaches(false);
>>>       return readURLConnectionAsBytes(conn);
>>>     } catch (IOException e) {
>>>       return null; // This causes NullPointerException, but this exception
>>>       isn't logged (wtf, GWT!)
>>>     }
>>>   }
>>>
>>> To summarize, I'll need to write another patch which replaces
>>> Util.readURLAsBytes implementation via AOP and prints IOException details.
>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> # CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml
>>>>>>>>>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M -XX:MaxPermSize=1024m
>>>>>>>>>> -Dgwt.compiler.localWorkers=1"
>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}"
>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D "ovirt_build_extra_flags
>>>>>>>>>> -gs
>>>>>>>>>> ${CI_MAVEN_SETTINGS} -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera
>>>>>>>>>> -Dgwt.logLevel=TRACE -X" --rebuild rpmbuild/SRPMS/*.src.rpm
>>>>>>>>>>
>>>>>>>>>> Let me know if you need anything else.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Vojtech
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>> From: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>>>>>> To: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>>>>>>>>> "arch" <arch at ovirt.org>
>>>>>>>>>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM
>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>>>>>>>
>>>>>>>>>>>> In your log I see following error:
>>>>>>>>>>>>
>>>>>>>>>>>> ...
>>>>>>>>>>>> [INFO]    Computing all possible rebind results for
>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
>>>>>>>>>>>> [INFO]       Rebinding
>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
>>>>>>>>>>>> [INFO]          Invoking generator
>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
>>>>>>>>>>>> [INFO]             [ERROR] Generator
>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
>>>>>>>>>>>> threw
>>>>>>>>>>>> an exception while rebinding
>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed
>>>>>>>>>>>> for
>>>>>>>>>>>> removeBrickCommitRequired
>>>>>>>>>>>> [INFO] 	at
>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
>>>>>>>>>>>> ...
>>>>>>>>>>>>
>>>>>>>>>>>> I checked again logs at
>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host/consoleFull
>>>>>>>>>>>> and indeed there is a similar error:
>>>>>>>>>>>>
>>>>>>>>>>>> ...
>>>>>>>>>>>> [INFO]    Computing all possible rebind results for
>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
>>>>>>>>>>>> [INFO]       Rebinding
>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
>>>>>>>>>>>> [INFO]          Invoking generator
>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
>>>>>>>>>>>> [INFO]             [ERROR] Generator
>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
>>>>>>>>>>>> threw
>>>>>>>>>>>> an exception while rebinding
>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
>>>>>>>>>>>> [INFO] java.lang.NullPointerException
>>>>>>>>>>>> [INFO] 	at
>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
>>>>>>>>>>>> ...
>>>>>>>>>>>>
>>>>>>>>>>>> NullPointerException when processing ApplicationResources but
>>>>>>>>>>>> occuring
>>>>>>>>>>>> from
>>>>>>>>>>>> different places (ImageResourceGenerator vs.
>>>>>>>>>>>> InlineResourceContext)
>>>>>>>>>>>>
>>>>>>>>>>>> Now I understand why Maven build fails without any details:
>>>>>>>>>>>> - GWT deferred binding fails for ApplicationResources (i.e.
>>>>>>>>>>>> uncaught
>>>>>>>>>>>> NPE)
>>>>>>>>>>>> - GWT compilation proceeds, fails to link with
>>>>>>>>>>>> ApplicationResources
>>>>>>>>>>>> - GWT compilation task fails with non-OK status code
>>>>>>>>>>>>
>>>>>>>>>>>> So the issue here is GWT deferred binding generator
>>>>>>>>>>>> (InlineClientBundleGenerator) failure during GWT compilation.
>>>>>>>>>>>>
>>>>>>>>>>>> GWT compiler writes extra Java files during deferred binding
>>>>>>>>>>>> phase, so this could point to issue with local environment..
>>>>>>>>>>>>
>>>>>>>>>>>> I will try to investigate this and look at GWT source code.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Sandro Bonazzola
>>>>>>>>>> Better technology. Faster innovation. Powered by community
>>>>>>>>>> collaboration.
>>>>>>>>>> See how it works at redhat.com
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Sandro Bonazzola
>>>>>>>> Better technology. Faster innovation. Powered by community
>>>>>>>> collaboration.
>>>>>>>> See how it works at redhat.com
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Sandro Bonazzola
>>>>>> Better technology. Faster innovation. Powered by community
>>>>>> collaboration.
>>>>>> See how it works at redhat.com
>>>>>>
>>>>
>>>>
>>>> --
>>>> Sandro Bonazzola
>>>> Better technology. Faster innovation. Powered by community collaboration.
>>>> See how it works at redhat.com
>>>>
>>
>>
>> --
>> Sandro Bonazzola
>> Better technology. Faster innovation. Powered by community collaboration.
>> See how it works at redhat.com
>>


-- 
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com

More information about the Arch mailing list