Jenkins issue (findbugs)

Thu Jun 5 15:26:00 UTC 2014

+1,

i think we should start looking into that.
https://code.google.com/p/gerrit/wiki/Scaling

eyal.

----- Original Message -----
> From: "David Caro" <dcaroest at redhat.com>
> To: "Eyal Edri" <eedri at redhat.com>
> Cc: infra at ovirt.org, "Michal Skrivanek" <michal.skrivanek at redhat.com>
> Sent: Thursday, June 5, 2014 6:20:12 PM
> Subject: Re: Jenkins issue (findbugs)
> 
> On Thu 05 Jun 2014 05:10:48 PM CEST, Eyal Edri wrote:
> > i think this happens when gerrit server is too overloaded, and can't return
> > an answer for more than 10min (default value).
> > latest git plugin allows us to set a different value than 10min.
> >
> > for 1st action, i'd change those failing jobs to use maybe 20min instead,
> > and see if it's fixed.
> > can you provide list of links for the failing jobs you saw?
> >
> > for longer run, i'd think on how to reduce/manage load on gerrit server,
> > maybe using zuul. [1]
> > david - any thoughts?
> >
> > [1] http://ci.openstack.org/zuul/
> >
> > ----- Original Message -----
> >> From: "Michal Skrivanek" <michal.skrivanek at redhat.com>
> >> To: infra at ovirt.org
> >> Sent: Thursday, June 5, 2014 5:36:59 PM
> >> Subject: Jenkins issue (findbugs)
> >>
> >>
> >>> ----- Forwarded Message -----
> >>> From: "Frantisek Kobzik" <fkobzik at redhat.com>
> >>> To: infra at ovirt.org
> >>> Sent: Thursday, June 5, 2014 11:23:12 AM
> >>> Subject: Jenkins issue (findbugs)
> >>>
> >>> Hi guys,
> >>>
> >>> I have an issue with jenkins build and I was advised to contact infra.
> >>> The console output is below (Similar issue also happens with other kinds
> >>> of
> >>> jobs).
> >>>
> >>> I'm seeing this issue quite often (~50 % of pushes), is there anything
> >>> you
> >>> can do about it?
> >>>
> >>> Thanks,
> >>> Franta.
> >>>
> >>>
> >>> 9:47:36 ERROR: Timeout after 10 minutes
> >>> 09:47:36 FATAL: Failed to fetch from git://gerrit.ovirt.org/ovirt-engine
> >>> 09:47:37 hudson.plugins.git.GitException: Failed to fetch from
> >>> git://gerrit.ovirt.org/ovirt-engine
> >>> 09:47:37 	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:623)
> >>> 09:47:37 	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:855)
> >>> 09:47:37 	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:880)
> >>> 09:47:37 	at
> >>> org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:118)
> >>> 09:47:37 	at
> >>> hudson.model.AbstractProject.checkout(AbstractProject.java:1320)
> >>> 09:47:37 	at
> >>> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:609)
> >>> 09:47:37 	at
> >>> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
> >>> 09:47:37 	at
> >>> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:518)
> >>> 09:47:37 	at hudson.model.Run.execute(Run.java:1700)
> >>> 09:47:37 	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> >>> 09:47:37 	at
> >>> hudson.model.ResourceController.execute(ResourceController.java:88)
> >>> 09:47:37 	at hudson.model.Executor.run(Executor.java:231)
> >>> 09:47:37 Caused by: hudson.plugins.git.GitException: Command "git fetch
> >>> --tags --progress git://gerrit.ovirt.org/ovirt-engine
> >>> refs/changes/14/28314/3 --prune" returned status code 143:
> >>> 09:47:37 stdout:
> >>> 9:47:37 stderr: remote: Counting objects: 136
> >>> remote: Counting objects: 375
> >>> remote: Counting objects: 582
> >>> .
> >>> etc.
> >>> .
> >>> 09:47:37 	at
> >>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1276)
> >>> 09:47:37 	at
> >>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1146)
> >>> 09:47:37 	at
> >>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImpl.java:87)
> >>> 09:47:37 	at
> >>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:254)
> >>> 09:47:37 	at
> >>> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
> >>> 09:47:37 	at
> >>> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
> >>> 09:47:37 	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
> >>> 09:47:37 	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
> >>> 09:47:37 	at hudson.remoting.Request$2.run(Request.java:328)
> >>> 09:47:37 	at
> >>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> >>> 09:47:37 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >>> 09:47:37 	at
> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>> 09:47:37 	at
> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>> 09:47:37 	at java.lang.Thread.run(Thread.java:744)
> >>> .
> >>
> >> _______________________________________________
> >> Infra mailing list
> >> Infra at ovirt.org
> >> http://lists.ovirt.org/mailman/listinfo/infra
> >>
> 
> Maybe we can start investigating on create mirrors for gerrit to pull
> from the slaves (also downstream if able) at least for read operations,
> that will leverage a lot of load from the jenkins slaves (if not all of
> it)
> 
> --
> David Caro
> 
> Red Hat S.L.
> Continuous Integration Engineer - EMEA ENG Virtualization R&D
> 
> Email: dcaro at redhat.com
> Web: www.redhat.com
> RHT Global #: 82-62605
> 
> 