Jenkins issue (findbugs)

[David Caro]

On Thu 05 Jun 2014 05:10:48 PM CEST, Eyal Edri wrote:
> i think this happens when gerrit server is too overloaded, and can't return an answer for more than 10min (default value).
> latest git plugin allows us to set a different value than 10min.
>
> for 1st action, i'd change those failing jobs to use maybe 20min instead, and see if it's fixed.
> can you provide list of links for the failing jobs you saw?
>
> for longer run, i'd think on how to reduce/manage load on gerrit server, maybe using zuul. [1]
> david - any thoughts?
>
> [1] http://ci.openstack.org/zuul/
>
> ----- Original Message -----
>> From: "Michal Skrivanek" <michal.skrivanek at redhat.com>
>> To: infra at ovirt.org
>> Sent: Thursday, June 5, 2014 5:36:59 PM
>> Subject: Jenkins issue (findbugs)
>>
>>
>>> ----- Forwarded Message -----
>>> From: "Frantisek Kobzik" <fkobzik at redhat.com>
>>> To: infra at ovirt.org
>>> Sent: Thursday, June 5, 2014 11:23:12 AM
>>> Subject: Jenkins issue (findbugs)
>>>
>>> Hi guys,
>>>
>>> I have an issue with jenkins build and I was advised to contact infra.
>>> The console output is below (Similar issue also happens with other kinds of
>>> jobs).
>>>
>>> I'm seeing this issue quite often (~50 % of pushes), is there anything you
>>> can do about it?
>>>
>>> Thanks,
>>> Franta.
>>>
>>>
>>> 9:47:36 ERROR: Timeout after 10 minutes
>>> 09:47:36 FATAL: Failed to fetch from git://gerrit.ovirt.org/ovirt-engine
>>> 09:47:37 hudson.plugins.git.GitException: Failed to fetch from
>>> git://gerrit.ovirt.org/ovirt-engine
>>> 09:47:37 	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:623)
>>> 09:47:37 	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:855)
>>> 09:47:37 	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:880)
>>> 09:47:37 	at
>>> org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:118)
>>> 09:47:37 	at
>>> hudson.model.AbstractProject.checkout(AbstractProject.java:1320)
>>> 09:47:37 	at
>>> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:609)
>>> 09:47:37 	at
>>> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
>>> 09:47:37 	at
>>> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:518)
>>> 09:47:37 	at hudson.model.Run.execute(Run.java:1700)
>>> 09:47:37 	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>>> 09:47:37 	at
>>> hudson.model.ResourceController.execute(ResourceController.java:88)
>>> 09:47:37 	at hudson.model.Executor.run(Executor.java:231)
>>> 09:47:37 Caused by: hudson.plugins.git.GitException: Command "git fetch
>>> --tags --progress git://gerrit.ovirt.org/ovirt-engine
>>> refs/changes/14/28314/3 --prune" returned status code 143:
>>> 09:47:37 stdout:
>>> 9:47:37 stderr: remote: Counting objects: 136
>>> remote: Counting objects: 375
>>> remote: Counting objects: 582
>>> .
>>> etc.
>>> .
>>> 09:47:37 	at
>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1276)
>>> 09:47:37 	at
>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1146)
>>> 09:47:37 	at
>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImpl.java:87)
>>> 09:47:37 	at
>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:254)
>>> 09:47:37 	at
>>> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
>>> 09:47:37 	at
>>> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
>>> 09:47:37 	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
>>> 09:47:37 	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
>>> 09:47:37 	at hudson.remoting.Request$2.run(Request.java:328)
>>> 09:47:37 	at
>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
>>> 09:47:37 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> 09:47:37 	at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> 09:47:37 	at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> 09:47:37 	at java.lang.Thread.run(Thread.java:744)
>>> .
>>
>> _______________________________________________
>> Infra mailing list
>> Infra at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/infra
>>

Maybe we can start investigating on create mirrors for gerrit to pull 
from the slaves (also downstream if able) at least for read operations, 
that will leverage a lot of load from the jenkins slaves (if not all of 
it)

--
David Caro

Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D

Email: dcaro at redhat.com
Web: www.redhat.com
RHT Global #: 82-62605

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20140605/10b2310a/attachment.sig>