+1, i think we should start looking into that. https://code.google.com/p/gerrit/wiki/Scaling eyal. ----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: infra@ovirt.org, "Michal Skrivanek" <michal.skrivanek@redhat.com> Sent: Thursday, June 5, 2014 6:20:12 PM Subject: Re: Jenkins issue (findbugs)
On Thu 05 Jun 2014 05:10:48 PM CEST, Eyal Edri wrote:
i think this happens when gerrit server is too overloaded, and can't return an answer for more than 10min (default value). latest git plugin allows us to set a different value than 10min.
for 1st action, i'd change those failing jobs to use maybe 20min instead, and see if it's fixed. can you provide list of links for the failing jobs you saw?
for longer run, i'd think on how to reduce/manage load on gerrit server, maybe using zuul. [1] david - any thoughts?
[1] http://ci.openstack.org/zuul/
----- Original Message -----
From: "Michal Skrivanek" <michal.skrivanek@redhat.com> To: infra@ovirt.org Sent: Thursday, June 5, 2014 5:36:59 PM Subject: Jenkins issue (findbugs)
----- Forwarded Message ----- From: "Frantisek Kobzik" <fkobzik@redhat.com> To: infra@ovirt.org Sent: Thursday, June 5, 2014 11:23:12 AM Subject: Jenkins issue (findbugs)
Hi guys,
I have an issue with jenkins build and I was advised to contact infra. The console output is below (Similar issue also happens with other kinds of jobs).
I'm seeing this issue quite often (~50 % of pushes), is there anything you can do about it?
Thanks, Franta.
9:47:36 ERROR: Timeout after 10 minutes 09:47:36 FATAL: Failed to fetch from git://gerrit.ovirt.org/ovirt-engine 09:47:37 hudson.plugins.git.GitException: Failed to fetch from git://gerrit.ovirt.org/ovirt-engine 09:47:37 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:623) 09:47:37 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:855) 09:47:37 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:880) 09:47:37 at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:118) 09:47:37 at hudson.model.AbstractProject.checkout(AbstractProject.java:1320) 09:47:37 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:609) 09:47:37 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88) 09:47:37 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:518) 09:47:37 at hudson.model.Run.execute(Run.java:1700) 09:47:37 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) 09:47:37 at hudson.model.ResourceController.execute(ResourceController.java:88) 09:47:37 at hudson.model.Executor.run(Executor.java:231) 09:47:37 Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress git://gerrit.ovirt.org/ovirt-engine refs/changes/14/28314/3 --prune" returned status code 143: 09:47:37 stdout: 9:47:37 stderr: remote: Counting objects: 136 remote: Counting objects: 375 remote: Counting objects: 582 . etc. . 09:47:37 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1276) 09:47:37 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1146) 09:47:37 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImpl.java:87) 09:47:37 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:254) 09:47:37 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) 09:47:37 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) 09:47:37 at hudson.remoting.UserRequest.perform(UserRequest.java:118) 09:47:37 at hudson.remoting.UserRequest.perform(UserRequest.java:48) 09:47:37 at hudson.remoting.Request$2.run(Request.java:328) 09:47:37 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) 09:47:37 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 09:47:37 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 09:47:37 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 09:47:37 at java.lang.Thread.run(Thread.java:744) .
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Maybe we can start investigating on create mirrors for gerrit to pull from the slaves (also downstream if able) at least for read operations, that will leverage a lot of load from the jenkins slaves (if not all of it)
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605