[urgent] Re: ovirt-engine build failure

Sandro Bonazzola sbonazzo at redhat.com
Wed Feb 26 07:37:07 UTC 2014 

Il 25/02/2014 16:52, Vojtech Szocs ha scritto:
> 
> 
> ----- Original Message -----
>> From: "Vojtech Szocs" <vszocs at redhat.com>
>> To: "Sandro Bonazzola" <sbonazzo at redhat.com>
>> Cc: "Juan Hernandez" <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>, "arch" <arch at ovirt.org>
>> Sent: Tuesday, February 25, 2014 1:05:35 PM
>> Subject: Re: [urgent] Re: ovirt-engine build failure
>>
>>
>>
>> ----- Original Message -----
>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>> Cc: "Juan Hernandez" <jhernand at redhat.com>, "arch" <arch at ovirt.org>, "Kiril
>>> Nesenko" <knesenko at redhat.com>
>>> Sent: Tuesday, February 25, 2014 9:02:11 AM
>>> Subject: Re: [urgent] Re: ovirt-engine build failure
>>>
>>> Il 25/02/2014 08:47, Sandro Bonazzola ha scritto:
>>>> Il 24/02/2014 18:28, Vojtech Szocs ha scritto:
>>>>>
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>>> "arch" <arch at ovirt.org>
>>>>>> Sent: Monday, February 24, 2014 9:08:09 AM
>>>>>> Subject: [urgent] Re: ovirt-engine build failure
>>>>>>
>>>>>> Top posting,
>>>>>>
>>>>>> Please give this task really urgent priority, as you can see:
>>>>>>
>>>>>> http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/
>>>>>> and
>>>>>> http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/
>>>>>>
>>>>>> are failing consistently so we don't have nightly builds.
>>>>>
>>>>> (CC'ing Einav)
>>>>>
>>>>> Attached TEST patch, please apply & rebuild ovirt-engine as usual.
>>>>>
>>>>> This patch should cause following output during GWT compilation:
>>>>>
>>>>> [ERROR] ######### Util.readURLAsBytes
>>>>> file:/home/vszocs/work/ovirt-engine/frontend/webadmin/modules/webadmin/target/generated-sources/gwt/org/ovirt/engine/ui/webadmin/images/search_button.png
>>>>>
>>>>> If IOException occurs, exception stacktrace will be printed right below
>>>>> that line.
>>>>>
>>>>> Note: please ignore the [ERROR] prefix, it's a technical issue, more
>>>>> details at
>>>>> https://www.mail-archive.com/engine-devel@ovirt.org/msg05735.html
>>>>>
>>>>> I just did a local upstream build with TEST patch applied, GWT
>>>>> compilation
>>>>> passed OK.
>>>>
>>>> build job running with your patch here:
>>>> http://jenkins.ovirt.org/job/manual-build-tarball/235/
>>>> waiting for a free executor right now.
>>>> Let's see how it goes.
>>>
>>> Failed on CentOS:
>>> http://jenkins.ovirt.org/job/manual-build-tarball/235/label=centos6-host/consoleFull
>>
>> Hm, in that ^^ build, following exceptions occured:
>>
>> 1.
>> [INFO] java.lang.NullPointerException
>> [INFO]  at com.google.gwt.dev.util.Util.computeStrongName(Util.java:172)
>>
>> 2.
>> [INFO] java.lang.NullPointerException: No ImageRect ever computed for
>> downDisabledImage
>> [INFO]  at
>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475
>>
>> 1. is caused by Util#readURLAsBytes but I don't see any IOException in the
>> logs
>> 2. is just follow-up consequence of 1. (i.e. image metadata missing in GWT
>> image cache)
>>
>> Now I noticed I missed one possible IOException inside
>> Util#readURLConnectionAsBytes ...
>>
>> Long story short, I'll send updated TEST patch soon, apologies for the
>> inconvenience.
> 
> TEST-v2 patch attached. Please apply & rebuild ovirt-engine as usual.

Done, here is the console output:
http://jenkins.ovirt.org/job/manual-build-tarball/label=centos6-host/239/consoleFull




> 
>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Il 20/02/2014 13:34, Vojtech Szocs ha scritto:
>>>>>>>
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>>>>> "arch" <arch at ovirt.org>
>>>>>>>> Sent: Thursday, February 20, 2014 10:02:00 AM
>>>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>>>
>>>>>>>> Il 19/02/2014 16:51, Vojtech Szocs ha scritto:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----- Original Message -----
>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>>>>>>> "arch" <arch at ovirt.org>
>>>>>>>>>> Sent: Wednesday, February 19, 2014 2:43:44 PM
>>>>>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>>>>>
>>>>>>>>>> Il 19/02/2014 14:41, Vojtech Szocs ha scritto:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>>>>>>>>> "arch" <arch at ovirt.org>
>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 2:23:16 PM
>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>>>>>>>
>>>>>>>>>>>> Il 19/02/2014 14:19, Vojtech Szocs ha scritto:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>>>>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>>>>>>>>>>> "arch" <arch at ovirt.org>
>>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 1:18:28 PM
>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto:
>>>>>>>>>>>>>>> Hi, I checked GWT sources:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever
>>>>>>>>>>>>>>>> computed
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> removeBrickCommitRequired
>>>>>>>>>>>>>>>> [INFO] 	at
>>>>>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This happened because internal image cache for bundled
>>>>>>>>>>>>>>> (composite
>>>>>>>>>>>>>>> PNG)
>>>>>>>>>>>>>>> images was corrput.
>>>>>>>>>>>>>>> It seems to be caused by disk I/O error while reading file
>>>>>>>>>>>>>>> images/glusterAsyncTasks/remove_brick_commit_required.png
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException
>>>>>>>>>>>>>>>> [INFO] 	at
>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This happened because image data was null, again possible disk
>>>>>>>>>>>>>>> I/O
>>>>>>>>>>>>>>> error
>>>>>>>>>>>>>>> while reading image file.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can we enable "TRACE" logging during GWT compilation?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>   -Dgwt.logLevel=TRACE
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Building master right now with your suggestions here:
>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/
>>>>>>>>>>>>>
>>>>>>>>>>>>> Looking at
>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host/console
>>>>>>>>>>>>>
>>>>>>>>>>>>> So far I see no errors.. There should be total 5 browsers x 8
>>>>>>>>>>>>> locales
>>>>>>>>>>>>> =
>>>>>>>>>>>>> 40
>>>>>>>>>>>>> GWT permutations.
>>>>>>>>>>>>
>>>>>>>>>>>> So no changes with and without
>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera.
>>>>>>>>>>>> We've still 40 permutations. Maybe you defined
>>>>>>>>>>>> userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the POM
>>>>>>>>>>>> files?
>>>>>>>>>>>
>>>>>>>>>>> You are right. I checked frontend/webadmin/modules/pom.xml -> the
>>>>>>>>>>> "gwt.userAgent"
>>>>>>>>>>> property is copied into "locale" GWT property in
>>>>>>>>>>> {WebAdmin,UserPortal}.gwt.xml
>>>>>>>>>>
>>>>>>>>>> Ok, so I'll remove -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera
>>>>>>>>>> from
>>>>>>>>>> build
>>>>>>>>>> command.
>>>>>>>>>
>>>>>>>>> Agreed.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This:
>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/console
>>>>>>>>>>>> failed, you can look at it, it has -Dgwt.logLevel=TRACE -X.
>>>>>>>>>>>
>>>>>>>>>>> Hm, it contains:
>>>>>>>>>>>
>>>>>>>>>>> [DEBUG]   (f) logLevel = INFO
>>>>>>>>>>>
>>>>>>>>>>> Somehow the TRACE log level for GWT wasn't applied.. not sure why.
>>>>>>>>>>> Let
>>>>>>>>>>> me
>>>>>>>>>>> check
>>>>>>>>>>> this on local build.
>>>>>>>>>>
>>>>>>>>>> Ok, let me know if I've to change command line.
>>>>>>>>>
>>>>>>>>> I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/
>>>>>>>>>
>>>>>>>>> When it's merged, setting "gwt.logLevel" will work (as you did
>>>>>>>>> before)
>>>>>>>>> and we'll be able to see more details in the build log.
>>>>>>>>
>>>>>>>> Here is the new build with your patch merged:
>>>>>>>>
>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/consoleFull
>>>>>>>
>>>>>>> Thanks. I did some more investigation on this.
>>>>>>>
>>>>>>> Again it's NullPointerException when rebinding ApplicationResources,
>>>>>>> but
>>>>>>> this time for different image file: ApplicationResources.wrenchImage
>>>>>>> ("images/wrench.png").
>>>>>>>
>>>>>>> Looking at stacktrace & GWT sources, the problematic code seems to be
>>>>>>> (AbstractResourceContext:74):
>>>>>>>
>>>>>>>   byte[] bytes = Util.readURLAsBytes(resource);
>>>>>>>
>>>>>>> where "resource" is (ImageResourceGenerator:311):
>>>>>>>
>>>>>>>   localized.getUrl(); // LocalizedImage localized
>>>>>>>
>>>>>>> Note: in oVirt UI we don't use localized images (i.e.
>>>>>>> "wrench_fr_FR.png")
>>>>>>> so "localized" in this case points to "wrench.png".
>>>>>>>
>>>>>>> So the problem seems to be reading this file:
>>>>>>>
>>>>>>>   frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png
>>>>>>>
>>>>>>> Since we're having many GWT permutations, this file ^^ (along with
>>>>>>> similar
>>>>>>> ones) is attempted to be read multiple times *from the same (single)
>>>>>>> GWT
>>>>>>> compiler Java process, before actual permutation worker Java
>>>>>>> sub-processes
>>>>>>> are spawned*. (More precisely, above NullPointerException happens at
>>>>>>> deferred binding phase which precedes actual GWT compilation per each
>>>>>>> permutation.)
>>>>>>>
>>>>>>> I think the root cause is Util.readURLAsBytes method used internally
>>>>>>> by
>>>>>>> GWT
>>>>>>> compiler:
>>>>>>>
>>>>>>>   public static byte[] readURLAsBytes(URL url) {
>>>>>>>     try {
>>>>>>>       URLConnection conn = url.openConnection();
>>>>>>>       conn.setUseCaches(false);
>>>>>>>       return readURLConnectionAsBytes(conn);
>>>>>>>     } catch (IOException e) {
>>>>>>>       return null; // This causes NullPointerException, but this
>>>>>>>       exception
>>>>>>>       isn't logged (wtf, GWT!)
>>>>>>>     }
>>>>>>>   }
>>>>>>>
>>>>>>> To summarize, I'll need to write another patch which replaces
>>>>>>> Util.readURLAsBytes implementation via AOP and prints IOException
>>>>>>> details.
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #
>>>>>>>>>>>>>> CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml
>>>>>>>>>>>>>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M
>>>>>>>>>>>>>> -XX:MaxPermSize=1024m
>>>>>>>>>>>>>> -Dgwt.compiler.localWorkers=1"
>>>>>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}"
>>>>>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D
>>>>>>>>>>>>>> "ovirt_build_extra_flags
>>>>>>>>>>>>>> -gs
>>>>>>>>>>>>>> ${CI_MAVEN_SETTINGS}
>>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera
>>>>>>>>>>>>>> -Dgwt.logLevel=TRACE -X" --rebuild rpmbuild/SRPMS/*.src.rpm
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Let me know if you need anything else.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Vojtech
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>>>>> From: "Vojtech Szocs" <vszocs at redhat.com>
>>>>>>>>>>>>>>>> To: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>>>>>>>>>>>>> Cc: "Eyal Edri" <eedri at redhat.com>, "Juan Hernandez"
>>>>>>>>>>>>>>>> <jhernand at redhat.com>, "Kiril Nesenko" <knesenko at redhat.com>,
>>>>>>>>>>>>>>>> "arch" <arch at ovirt.org>
>>>>>>>>>>>>>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM
>>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In your log I see following error:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>> [INFO]    Computing all possible rebind results for
>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
>>>>>>>>>>>>>>>> [INFO]       Rebinding
>>>>>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
>>>>>>>>>>>>>>>> [INFO]          Invoking generator
>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
>>>>>>>>>>>>>>>> [INFO]             [ERROR] Generator
>>>>>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
>>>>>>>>>>>>>>>> threw
>>>>>>>>>>>>>>>> an exception while rebinding
>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever
>>>>>>>>>>>>>>>> computed
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> removeBrickCommitRequired
>>>>>>>>>>>>>>>> [INFO] 	at
>>>>>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475)
>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I checked again logs at
>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host/consoleFull
>>>>>>>>>>>>>>>> and indeed there is a similar error:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>> [INFO]    Computing all possible rebind results for
>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
>>>>>>>>>>>>>>>> [INFO]       Rebinding
>>>>>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources
>>>>>>>>>>>>>>>> [INFO]          Invoking generator
>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator
>>>>>>>>>>>>>>>> [INFO]             [ERROR] Generator
>>>>>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator'
>>>>>>>>>>>>>>>> threw
>>>>>>>>>>>>>>>> an exception while rebinding
>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources'
>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException
>>>>>>>>>>>>>>>> [INFO] 	at
>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40)
>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> NullPointerException when processing ApplicationResources but
>>>>>>>>>>>>>>>> occuring
>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>> different places (ImageResourceGenerator vs.
>>>>>>>>>>>>>>>> InlineResourceContext)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Now I understand why Maven build fails without any details:
>>>>>>>>>>>>>>>> - GWT deferred binding fails for ApplicationResources (i.e.
>>>>>>>>>>>>>>>> uncaught
>>>>>>>>>>>>>>>> NPE)
>>>>>>>>>>>>>>>> - GWT compilation proceeds, fails to link with
>>>>>>>>>>>>>>>> ApplicationResources
>>>>>>>>>>>>>>>> - GWT compilation task fails with non-OK status code
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So the issue here is GWT deferred binding generator
>>>>>>>>>>>>>>>> (InlineClientBundleGenerator) failure during GWT compilation.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> GWT compiler writes extra Java files during deferred binding
>>>>>>>>>>>>>>>> phase, so this could point to issue with local environment..
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I will try to investigate this and look at GWT source code.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Sandro Bonazzola
>>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community
>>>>>>>>>>>>>> collaboration.
>>>>>>>>>>>>>> See how it works at redhat.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Sandro Bonazzola
>>>>>>>>>>>> Better technology. Faster innovation. Powered by community
>>>>>>>>>>>> collaboration.
>>>>>>>>>>>> See how it works at redhat.com
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Sandro Bonazzola
>>>>>>>>>> Better technology. Faster innovation. Powered by community
>>>>>>>>>> collaboration.
>>>>>>>>>> See how it works at redhat.com
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Sandro Bonazzola
>>>>>>>> Better technology. Faster innovation. Powered by community
>>>>>>>> collaboration.
>>>>>>>> See how it works at redhat.com
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Sandro Bonazzola
>>>>>> Better technology. Faster innovation. Powered by community
>>>>>> collaboration.
>>>>>> See how it works at redhat.com
>>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Sandro Bonazzola
>>> Better technology. Faster innovation. Powered by community collaboration.
>>> See how it works at redhat.com
>>>
>> _______________________________________________
>> Arch mailing list
>> Arch at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/arch
>>


-- 
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com

More information about the Arch mailing list