10:26 a.m.
+1,
i think we should start looking into that.
https://code.google.com/p/gerrit/wiki/Scaling
eyal.
----- Original Message -----
From: "David Caro" <dcaroest(a)redhat.com>
To: "Eyal Edri" <eedri(a)redhat.com>
Cc: infra(a)ovirt.org, "Michal Skrivanek" <michal.skrivanek(a)redhat.com>
Sent: Thursday, June 5, 2014 6:20:12 PM
Subject: Re: Jenkins issue (findbugs)
On Thu 05 Jun 2014 05:10:48 PM CEST, Eyal Edri wrote:
> i think this happens when gerrit server is too overloaded, and can't return
> an answer for more than 10min (default value).
> latest git plugin allows us to set a different value than 10min.
>
> for 1st action, i'd change those failing jobs to use maybe 20min instead,
> and see if it's fixed.
> can you provide list of links for the failing jobs you saw?
>
> for longer run, i'd think on how to reduce/manage load on gerrit server,
> maybe using zuul. [1]
> david - any thoughts?
>
> [1] http://ci.openstack.org/zuul/
>
> ----- Original Message -----
>> From: "Michal Skrivanek" <michal.skrivanek(a)redhat.com>
>> To: infra(a)ovirt.org
>> Sent: Thursday, June 5, 2014 5:36:59 PM
>> Subject: Jenkins issue (findbugs)
>>
>>
>>> ----- Forwarded Message -----
>>> From: "Frantisek Kobzik" <fkobzik(a)redhat.com>
>>> To: infra(a)ovirt.org
>>> Sent: Thursday, June 5, 2014 11:23:12 AM
>>> Subject: Jenkins issue (findbugs)
>>>
>>> Hi guys,
>>>
>>> I have an issue with jenkins build and I was advised to contact infra.
>>> The console output is below (Similar issue also happens with other kinds
>>> of
>>> jobs).
>>>
>>> I'm seeing this issue quite often (~50 % of pushes), is there anything
>>> you
>>> can do about it?
>>>
>>> Thanks,
>>> Franta.
>>>
>>>
>>> 9:47:36 ERROR: Timeout after 10 minutes
>>> 09:47:36 FATAL: Failed to fetch from git://gerrit.ovirt.org/ovirt-engine
>>> 09:47:37 hudson.plugins.git.GitException: Failed to fetch from
>>> git://gerrit.ovirt.org/ovirt-engine
>>> 09:47:37 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:623)
>>> 09:47:37 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:855)
>>> 09:47:37 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:880)
>>> 09:47:37 at
>>> org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:118)
>>> 09:47:37 at
>>> hudson.model.AbstractProject.checkout(AbstractProject.java:1320)
>>> 09:47:37 at
>>>
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:609)
>>> 09:47:37 at
>>> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
>>> 09:47:37 at
>>>
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:518)
>>> 09:47:37 at hudson.model.Run.execute(Run.java:1700)
>>> 09:47:37 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>>> 09:47:37 at
>>> hudson.model.ResourceController.execute(ResourceController.java:88)
>>> 09:47:37 at hudson.model.Executor.run(Executor.java:231)
>>> 09:47:37 Caused by: hudson.plugins.git.GitException: Command "git
fetch
>>> --tags --progress git://gerrit.ovirt.org/ovirt-engine
>>> refs/changes/14/28314/3 --prune" returned status code 143:
>>> 09:47:37 stdout:
>>> 9:47:37 stderr: remote: Counting objects: 136
>>> remote: Counting objects: 375
>>> remote: Counting objects: 582
>>> .
>>> etc.
>>> .
>>> 09:47:37 at
>>>
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1276)
>>> 09:47:37 at
>>>
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1146)
>>> 09:47:37 at
>>>
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImpl.java:87)
>>> 09:47:37 at
>>>
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:254)
>>> 09:47:37 at
>>>
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
>>> 09:47:37 at
>>>
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
>>> 09:47:37 at hudson.remoting.UserRequest.perform(UserRequest.java:118)
>>> 09:47:37 at hudson.remoting.UserRequest.perform(UserRequest.java:48)
>>> 09:47:37 at hudson.remoting.Request$2.run(Request.java:328)
>>> 09:47:37 at
>>>
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
>>> 09:47:37 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> 09:47:37 at
>>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> 09:47:37 at
>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> 09:47:37 at java.lang.Thread.run(Thread.java:744)
>>> .
>>
>> _______________________________________________
>> Infra mailing list
>> Infra(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/infra
>>
Maybe we can start investigating on create mirrors for gerrit to pull
from the slaves (also downstream if able) at least for read operations,
that will leverage a lot of load from the jenkins slaves (if not all of
it)
--
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Email: dcaro(a)redhat.com
Web: www.redhat.com
RHT Global #: 82-62605