----- Forwarded Message ----- From: "Frantisek Kobzik" <fkobzik@redhat.com> To: infra@ovirt.org Sent: Thursday, June 5, 2014 11:23:12 AM Subject: Jenkins issue (findbugs)
Hi guys,
I have an issue with jenkins build and I was advised to contact infra. The console output is below (Similar issue also happens with other kinds of jobs).
I'm seeing this issue quite often (~50 % of pushes), is there anything you can do about it?
Thanks, Franta.
9:47:36 ERROR: Timeout after 10 minutes 09:47:36 FATAL: Failed to fetch from git://gerrit.ovirt.org/ovirt-engine 09:47:37 hudson.plugins.git.GitException: Failed to fetch from git://gerrit.ovirt.org/ovirt-engine 09:47:37 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:623) 09:47:37 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:855) 09:47:37 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:880) 09:47:37 at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:118) 09:47:37 at hudson.model.AbstractProject.checkout(AbstractProject.java:1320) 09:47:37 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:609) 09:47:37 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88) 09:47:37 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:518) 09:47:37 at hudson.model.Run.execute(Run.java:1700) 09:47:37 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) 09:47:37 at hudson.model.ResourceController.execute(ResourceController.java:88) 09:47:37 at hudson.model.Executor.run(Executor.java:231) 09:47:37 Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress git://gerrit.ovirt.org/ovirt-engine refs/changes/14/28314/3 --prune" returned status code 143: 09:47:37 stdout: 9:47:37 stderr: remote: Counting objects: 136 remote: Counting objects: 375 remote: Counting objects: 582 . etc. . 09:47:37 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1276) 09:47:37 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1146) 09:47:37 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImpl.java:87) 09:47:37 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:254) 09:47:37 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) 09:47:37 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) 09:47:37 at hudson.remoting.UserRequest.perform(UserRequest.java:118) 09:47:37 at hudson.remoting.UserRequest.perform(UserRequest.java:48) 09:47:37 at hudson.remoting.Request$2.run(Request.java:328) 09:47:37 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) 09:47:37 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 09:47:37 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 09:47:37 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 09:47:37 at java.lang.Thread.run(Thread.java:744) .
i think this happens when gerrit server is too overloaded, and can't return an answer for more than 10min (default value). latest git plugin allows us to set a different value than 10min. for 1st action, i'd change those failing jobs to use maybe 20min instead, and see if it's fixed. can you provide list of links for the failing jobs you saw? for longer run, i'd think on how to reduce/manage load on gerrit server, maybe using zuul. [1] david - any thoughts? [1] http://ci.openstack.org/zuul/ ----- Original Message -----
From: "Michal Skrivanek" <michal.skrivanek@redhat.com> To: infra@ovirt.org Sent: Thursday, June 5, 2014 5:36:59 PM Subject: Jenkins issue (findbugs)
----- Forwarded Message ----- From: "Frantisek Kobzik" <fkobzik@redhat.com> To: infra@ovirt.org Sent: Thursday, June 5, 2014 11:23:12 AM Subject: Jenkins issue (findbugs)
Hi guys,
I have an issue with jenkins build and I was advised to contact infra. The console output is below (Similar issue also happens with other kinds of jobs).
I'm seeing this issue quite often (~50 % of pushes), is there anything you can do about it?
Thanks, Franta.
9:47:36 ERROR: Timeout after 10 minutes 09:47:36 FATAL: Failed to fetch from git://gerrit.ovirt.org/ovirt-engine 09:47:37 hudson.plugins.git.GitException: Failed to fetch from git://gerrit.ovirt.org/ovirt-engine 09:47:37 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:623) 09:47:37 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:855) 09:47:37 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:880) 09:47:37 at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:118) 09:47:37 at hudson.model.AbstractProject.checkout(AbstractProject.java:1320) 09:47:37 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:609) 09:47:37 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88) 09:47:37 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:518) 09:47:37 at hudson.model.Run.execute(Run.java:1700) 09:47:37 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) 09:47:37 at hudson.model.ResourceController.execute(ResourceController.java:88) 09:47:37 at hudson.model.Executor.run(Executor.java:231) 09:47:37 Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress git://gerrit.ovirt.org/ovirt-engine refs/changes/14/28314/3 --prune" returned status code 143: 09:47:37 stdout: 9:47:37 stderr: remote: Counting objects: 136 remote: Counting objects: 375 remote: Counting objects: 582 . etc. . 09:47:37 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1276) 09:47:37 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1146) 09:47:37 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImpl.java:87) 09:47:37 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:254) 09:47:37 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) 09:47:37 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) 09:47:37 at hudson.remoting.UserRequest.perform(UserRequest.java:118) 09:47:37 at hudson.remoting.UserRequest.perform(UserRequest.java:48) 09:47:37 at hudson.remoting.Request$2.run(Request.java:328) 09:47:37 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) 09:47:37 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 09:47:37 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 09:47:37 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 09:47:37 at java.lang.Thread.run(Thread.java:744) .
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
i think this happens when gerrit server is too overloaded, and can't re= turn an answer for more than 10min (default value). latest git plugin allows us to set a different value than 10min.
for 1st action, i'd change those failing jobs to use maybe 20min instea= d, and see if it's fixed. can you provide list of links for the failing jobs you saw?
for longer run, i'd think on how to reduce/manage load on gerrit server= , maybe using zuul. [1] david - any thoughts?
[1] http://ci.openstack.org/zuul/
----- Original Message -----
From: "Michal Skrivanek" <michal.skrivanek@redhat.com> To: infra@ovirt.org Sent: Thursday, June 5, 2014 5:36:59 PM Subject: Jenkins issue (findbugs)
----- Forwarded Message ----- From: "Frantisek Kobzik" <fkobzik@redhat.com> To: infra@ovirt.org Sent: Thursday, June 5, 2014 11:23:12 AM Subject: Jenkins issue (findbugs)
Hi guys,
I have an issue with jenkins build and I was advised to contact infra= =2E The console output is below (Similar issue also happens with other ki= nds of jobs).
I'm seeing this issue quite often (~50 % of pushes), is there anythin= g you can do about it?
Thanks, Franta.
9:47:36 ERROR: Timeout after 10 minutes 09:47:36 FATAL: Failed to fetch from git://gerrit.ovirt.org/ovirt-eng= ine 09:47:37 hudson.plugins.git.GitException: Failed to fetch from git://gerrit.ovirt.org/ovirt-engine 09:47:37 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:623) 09:47:37 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:85=
09:47:37 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:880) 09:47:37 at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:11=
09:47:37 at hudson.model.AbstractProject.checkout(AbstractProject.java:1320) 09:47:37 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(Abs=
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --57GaMnHJUE8iAksGB9GERqTGcdbIQoH4D Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Thu 05 Jun 2014 05:10:48 PM CEST, Eyal Edri wrote: 5) 8) tractBuild.java:609)
09:47:37 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)=
09:47:37 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.j= ava:518) 09:47:37 at hudson.model.Run.execute(Run.java:1700) 09:47:37 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) 09:47:37 at hudson.model.ResourceController.execute(ResourceController.java:88) 09:47:37 at hudson.model.Executor.run(Executor.java:231) 09:47:37 Caused by: hudson.plugins.git.GitException: Command "git fet= ch --tags --progress git://gerrit.ovirt.org/ovirt-engine refs/changes/14/28314/3 --prune" returned status code 143: 09:47:37 stdout: 9:47:37 stderr: remote: Counting objects: 136 remote: Counting objects: 375 remote: Counting objects: 582 . etc. . 09:47:37 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitA= PIImpl.java:1276) 09:47:37 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCreden= tials(CliGitAPIImpl.java:1146) 09:47:37 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImp= l.java:87) 09:47:37 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl= =2Ejava:254) 09:47:37 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandle= r$1.call(RemoteGitImpl.java:153) 09:47:37 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandle= r$1.call(RemoteGitImpl.java:146) 09:47:37 at hudson.remoting.UserRequest.perform(UserRequest.java:118= ) 09:47:37 at hudson.remoting.UserRequest.perform(UserRequest.java:48)=
09:47:37 at hudson.remoting.Request$2.run(Request.java:328) 09:47:37 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecut= orService.java:72) 09:47:37 at java.util.concurrent.FutureTask.run(FutureTask.java:262)=
09:47:37 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.= java:1145) 09:47:37 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= =2Ejava:615) 09:47:37 at java.lang.Thread.run(Thread.java:744) .
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Maybe we can start investigating on create mirrors for gerrit to pull=20 from the slaves (also downstream if able) at least for read operations,=20 that will leverage a lot of load from the jenkins slaves (if not all of=20 it) -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --57GaMnHJUE8iAksGB9GERqTGcdbIQoH4D Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJTkIqsAAoJEEBxx+HSYmnD53wH/jdr2N5aY3R0ESVaVp80IyBb 0g/jlvBY3BoKe1fdpWh0PCXRkI/uh3sUGO0NmVD1KHJgXd8+00RlZYnh/HIQrdTf sQzkIs5J54WQusvVA1CXeMJdWEPhDWv4+haeS+DtsITj+5VI86q8WP9siYpRy26j v3WmHeEBBxBCmHZNtKi0Nu0GG7/5IcBtFneyxpDBCvrs9hbe+BS14ScczaCAC1nq NVEUrC5h7tLPDxblXzu7hbrbUxFYj341D9jMrgQgYFIpnYB+WVn8XKB6jDK0GoV1 y0OAO1pSRvAli2BS/zxkSm2LemktmVh8k6o1odKb5katzq67ZhCtthPBgqAL3zg= =GhCF -----END PGP SIGNATURE----- --57GaMnHJUE8iAksGB9GERqTGcdbIQoH4D--
+1, i think we should start looking into that. https://code.google.com/p/gerrit/wiki/Scaling eyal. ----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: infra@ovirt.org, "Michal Skrivanek" <michal.skrivanek@redhat.com> Sent: Thursday, June 5, 2014 6:20:12 PM Subject: Re: Jenkins issue (findbugs)
On Thu 05 Jun 2014 05:10:48 PM CEST, Eyal Edri wrote:
i think this happens when gerrit server is too overloaded, and can't return an answer for more than 10min (default value). latest git plugin allows us to set a different value than 10min.
for 1st action, i'd change those failing jobs to use maybe 20min instead, and see if it's fixed. can you provide list of links for the failing jobs you saw?
for longer run, i'd think on how to reduce/manage load on gerrit server, maybe using zuul. [1] david - any thoughts?
[1] http://ci.openstack.org/zuul/
----- Original Message -----
From: "Michal Skrivanek" <michal.skrivanek@redhat.com> To: infra@ovirt.org Sent: Thursday, June 5, 2014 5:36:59 PM Subject: Jenkins issue (findbugs)
----- Forwarded Message ----- From: "Frantisek Kobzik" <fkobzik@redhat.com> To: infra@ovirt.org Sent: Thursday, June 5, 2014 11:23:12 AM Subject: Jenkins issue (findbugs)
Hi guys,
I have an issue with jenkins build and I was advised to contact infra. The console output is below (Similar issue also happens with other kinds of jobs).
I'm seeing this issue quite often (~50 % of pushes), is there anything you can do about it?
Thanks, Franta.
9:47:36 ERROR: Timeout after 10 minutes 09:47:36 FATAL: Failed to fetch from git://gerrit.ovirt.org/ovirt-engine 09:47:37 hudson.plugins.git.GitException: Failed to fetch from git://gerrit.ovirt.org/ovirt-engine 09:47:37 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:623) 09:47:37 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:855) 09:47:37 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:880) 09:47:37 at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:118) 09:47:37 at hudson.model.AbstractProject.checkout(AbstractProject.java:1320) 09:47:37 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:609) 09:47:37 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88) 09:47:37 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:518) 09:47:37 at hudson.model.Run.execute(Run.java:1700) 09:47:37 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) 09:47:37 at hudson.model.ResourceController.execute(ResourceController.java:88) 09:47:37 at hudson.model.Executor.run(Executor.java:231) 09:47:37 Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress git://gerrit.ovirt.org/ovirt-engine refs/changes/14/28314/3 --prune" returned status code 143: 09:47:37 stdout: 9:47:37 stderr: remote: Counting objects: 136 remote: Counting objects: 375 remote: Counting objects: 582 . etc. . 09:47:37 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1276) 09:47:37 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1146) 09:47:37 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImpl.java:87) 09:47:37 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:254) 09:47:37 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) 09:47:37 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) 09:47:37 at hudson.remoting.UserRequest.perform(UserRequest.java:118) 09:47:37 at hudson.remoting.UserRequest.perform(UserRequest.java:48) 09:47:37 at hudson.remoting.Request$2.run(Request.java:328) 09:47:37 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) 09:47:37 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 09:47:37 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 09:47:37 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 09:47:37 at java.lang.Thread.run(Thread.java:744) .
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Maybe we can start investigating on create mirrors for gerrit to pull from the slaves (also downstream if able) at least for read operations, that will leverage a lot of load from the jenkins slaves (if not all of it)
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
participants (3)
-
David Caro -
Eyal Edri -
Michal Skrivanek