9:36 a.m.
----- Forwarded Message -----
From: "Frantisek Kobzik" <fkobzik(a)redhat.com>
To: infra(a)ovirt.org
Sent: Thursday, June 5, 2014 11:23:12 AM
Subject: Jenkins issue (findbugs)
Hi guys,
I have an issue with jenkins build and I was advised to contact infra.
The console output is below (Similar issue also happens with other kinds of jobs).
I'm seeing this issue quite often (~50 % of pushes), is there anything you can do
about it?
Thanks,
Franta.
9:47:36 ERROR: Timeout after 10 minutes
09:47:36 FATAL: Failed to fetch from git://gerrit.ovirt.org/ovirt-engine
09:47:37 hudson.plugins.git.GitException: Failed to fetch from
git://gerrit.ovirt.org/ovirt-engine
09:47:37 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:623)
09:47:37 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:855)
09:47:37 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:880)
09:47:37 at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:118)
09:47:37 at hudson.model.AbstractProject.checkout(AbstractProject.java:1320)
09:47:37 at
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:609)
09:47:37 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
09:47:37 at
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:518)
09:47:37 at hudson.model.Run.execute(Run.java:1700)
09:47:37 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
09:47:37 at hudson.model.ResourceController.execute(ResourceController.java:88)
09:47:37 at hudson.model.Executor.run(Executor.java:231)
09:47:37 Caused by: hudson.plugins.git.GitException: Command "git fetch --tags
--progress git://gerrit.ovirt.org/ovirt-engine refs/changes/14/28314/3 --prune"
returned status code 143:
09:47:37 stdout:
9:47:37 stderr: remote: Counting objects: 136
remote: Counting objects: 375
remote: Counting objects: 582
.
etc.
.
09:47:37 at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1276)
09:47:37 at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1146)
09:47:37 at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImpl.java:87)
09:47:37 at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:254)
09:47:37 at
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
09:47:37 at
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
09:47:37 at hudson.remoting.UserRequest.perform(UserRequest.java:118)
09:47:37 at hudson.remoting.UserRequest.perform(UserRequest.java:48)
09:47:37 at hudson.remoting.Request$2.run(Request.java:328)
09:47:37 at
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
09:47:37 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
09:47:37 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
09:47:37 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
09:47:37 at java.lang.Thread.run(Thread.java:744)
.
10:10 a.m.
i think this happens when gerrit server is too overloaded, and can't return an answer
for more than 10min (default value).
latest git plugin allows us to set a different value than 10min.
for 1st action, i'd change those failing jobs to use maybe 20min instead, and see if
it's fixed.
can you provide list of links for the failing jobs you saw?
for longer run, i'd think on how to reduce/manage load on gerrit server, maybe using
zuul. [1]
david - any thoughts?
[1] http://ci.openstack.org/zuul/
----- Original Message -----
From: "Michal Skrivanek"
<michal.skrivanek(a)redhat.com>
To: infra(a)ovirt.org
Sent: Thursday, June 5, 2014 5:36:59 PM
Subject: Jenkins issue (findbugs)
> ----- Forwarded Message -----
> From: "Frantisek Kobzik" <fkobzik(a)redhat.com>
> To: infra(a)ovirt.org
> Sent: Thursday, June 5, 2014 11:23:12 AM
> Subject: Jenkins issue (findbugs)
>
> Hi guys,
>
> I have an issue with jenkins build and I was advised to contact infra.
> The console output is below (Similar issue also happens with other kinds of
> jobs).
>
> I'm seeing this issue quite often (~50 % of pushes), is there anything you
> can do about it?
>
> Thanks,
> Franta.
>
>
> 9:47:36 ERROR: Timeout after 10 minutes
> 09:47:36 FATAL: Failed to fetch from git://gerrit.ovirt.org/ovirt-engine
> 09:47:37 hudson.plugins.git.GitException: Failed to fetch from
> git://gerrit.ovirt.org/ovirt-engine
> 09:47:37 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:623)
> 09:47:37 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:855)
> 09:47:37 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:880)
> 09:47:37 at
> org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:118)
> 09:47:37 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1320)
> 09:47:37 at
>
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:609)
> 09:47:37 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
> 09:47:37 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:518)
> 09:47:37 at hudson.model.Run.execute(Run.java:1700)
> 09:47:37 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 09:47:37 at
> hudson.model.ResourceController.execute(ResourceController.java:88)
> 09:47:37 at hudson.model.Executor.run(Executor.java:231)
> 09:47:37 Caused by: hudson.plugins.git.GitException: Command "git fetch
> --tags --progress git://gerrit.ovirt.org/ovirt-engine
> refs/changes/14/28314/3 --prune" returned status code 143:
> 09:47:37 stdout:
> 9:47:37 stderr: remote: Counting objects: 136
> remote: Counting objects: 375
> remote: Counting objects: 582
> .
> etc.
> .
> 09:47:37 at
>
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1276)
> 09:47:37 at
>
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1146)
> 09:47:37 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImpl.java:87)
> 09:47:37 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:254)
> 09:47:37 at
>
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
> 09:47:37 at
>
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
> 09:47:37 at hudson.remoting.UserRequest.perform(UserRequest.java:118)
> 09:47:37 at hudson.remoting.UserRequest.perform(UserRequest.java:48)
> 09:47:37 at hudson.remoting.Request$2.run(Request.java:328)
> 09:47:37 at
>
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 09:47:37 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 09:47:37 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 09:47:37 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 09:47:37 at java.lang.Thread.run(Thread.java:744)
> .
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
10:20 a.m.
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--57GaMnHJUE8iAksGB9GERqTGcdbIQoH4D
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
On Thu 05 Jun 2014 05:10:48 PM CEST, Eyal Edri wrote:
i think this happens when gerrit server is too overloaded, and
can't re=
turn an answer for more than 10min (default value).
latest git plugin allows us to set a different value than 10min.
for 1st action, i'd change those failing jobs to use maybe 20min instea=
d, and
see if it's fixed.
can you provide list of links for the failing jobs you saw?
for longer run, i'd think on how to reduce/manage load on gerrit server=
,
maybe using zuul. [1]
david - any thoughts?
[1] http://ci.openstack.org/zuul/
----- Original Message -----
> From: "Michal Skrivanek" <michal.skrivanek(a)redhat.com>
> To: infra(a)ovirt.org
> Sent: Thursday, June 5, 2014 5:36:59 PM
> Subject: Jenkins issue (findbugs)
>
>
>> ----- Forwarded Message -----
>> From: "Frantisek Kobzik" <fkobzik(a)redhat.com>
>> To: infra(a)ovirt.org
>> Sent: Thursday, June 5, 2014 11:23:12 AM
>> Subject: Jenkins issue (findbugs)
>>
>> Hi guys,
>>
>> I have an issue with jenkins build and I was advised to contact infra=
=2E
>> The console output is below (Similar issue also happens with
other ki=
nds of
>> jobs).
>>
>> I'm seeing this issue quite often (~50 % of pushes), is there anythin=
g you
>> can do about it?
>>
>> Thanks,
>> Franta.
>>
>>
>> 9:47:36 ERROR: Timeout after 10 minutes
>> 09:47:36 FATAL: Failed to fetch from git://gerrit.ovirt.org/ovirt-eng=
ine
>> 09:47:37 hudson.plugins.git.GitException: Failed to fetch
from
>> git://gerrit.ovirt.org/ovirt-engine
>> 09:47:37 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:623)
>> 09:47:37 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:85=
5)
>> 09:47:37 at
hudson.plugins.git.GitSCM.checkout(GitSCM.java:880)
>> 09:47:37 at
>> org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:11=
8)
>> 09:47:37 at
>> hudson.model.AbstractProject.checkout(AbstractProject.java:1320)
>> 09:47:37 at
>> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(Abs=
tractBuild.java:609)
>> 09:47:37 at
>> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)=
>> 09:47:37 at
>> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.j=
ava:518)
>> 09:47:37 at hudson.model.Run.execute(Run.java:1700)
>> 09:47:37 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>> 09:47:37 at
>> hudson.model.ResourceController.execute(ResourceController.java:88)
>> 09:47:37 at hudson.model.Executor.run(Executor.java:231)
>> 09:47:37 Caused by: hudson.plugins.git.GitException: Command "git fet=
ch
>> --tags --progress git://gerrit.ovirt.org/ovirt-engine
>> refs/changes/14/28314/3 --prune" returned status code 143:
>> 09:47:37 stdout:
>> 9:47:37 stderr: remote: Counting objects: 136
>> remote: Counting objects: 375
>> remote: Counting objects: 582
>> .
>> etc.
>> .
>> 09:47:37 at
>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitA=
PIImpl.java:1276)
>> 09:47:37 at
>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCreden=
tials(CliGitAPIImpl.java:1146)
>> 09:47:37 at
>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImp=
l.java:87)
>> 09:47:37 at
>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl=
=2Ejava:254)
>> 09:47:37 at
>> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandle=
r$1.call(RemoteGitImpl.java:153)
>> 09:47:37 at
>> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandle=
r$1.call(RemoteGitImpl.java:146)
>> 09:47:37 at
hudson.remoting.UserRequest.perform(UserRequest.java:118=
)
>> 09:47:37 at
hudson.remoting.UserRequest.perform(UserRequest.java:48)=
>> 09:47:37 at hudson.remoting.Request$2.run(Request.java:328)
>> 09:47:37 at
>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecut=
orService.java:72)
>> 09:47:37 at
java.util.concurrent.FutureTask.run(FutureTask.java:262)=
>> 09:47:37 at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.=
java:1145)
>> 09:47:37 at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor=
=2Ejava:615)
>> 09:47:37 at java.lang.Thread.run(Thread.java:744)
>> .
>
> _______________________________________________
> Infra mailing list
> Infra(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>
Maybe we can start investigating on create mirrors for gerrit to pull=20
from the slaves (also downstream if able) at least for read operations,=20
that will leverage a lot of load from the jenkins slaves (if not all of=20
it)
--
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Email: dcaro(a)redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
--57GaMnHJUE8iAksGB9GERqTGcdbIQoH4D
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJTkIqsAAoJEEBxx+HSYmnD53wH/jdr2N5aY3R0ESVaVp80IyBb
0g/jlvBY3BoKe1fdpWh0PCXRkI/uh3sUGO0NmVD1KHJgXd8+00RlZYnh/HIQrdTf
sQzkIs5J54WQusvVA1CXeMJdWEPhDWv4+haeS+DtsITj+5VI86q8WP9siYpRy26j
v3WmHeEBBxBCmHZNtKi0Nu0GG7/5IcBtFneyxpDBCvrs9hbe+BS14ScczaCAC1nq
NVEUrC5h7tLPDxblXzu7hbrbUxFYj341D9jMrgQgYFIpnYB+WVn8XKB6jDK0GoV1
y0OAO1pSRvAli2BS/zxkSm2LemktmVh8k6o1odKb5katzq67ZhCtthPBgqAL3zg=
=GhCF
-----END PGP SIGNATURE-----
--57GaMnHJUE8iAksGB9GERqTGcdbIQoH4D--
10:26 a.m.
+1,
i think we should start looking into that.
https://code.google.com/p/gerrit/wiki/Scaling
eyal.
----- Original Message -----
From: "David Caro" <dcaroest(a)redhat.com>
To: "Eyal Edri" <eedri(a)redhat.com>
Cc: infra(a)ovirt.org, "Michal Skrivanek" <michal.skrivanek(a)redhat.com>
Sent: Thursday, June 5, 2014 6:20:12 PM
Subject: Re: Jenkins issue (findbugs)
On Thu 05 Jun 2014 05:10:48 PM CEST, Eyal Edri wrote:
> i think this happens when gerrit server is too overloaded, and can't return
> an answer for more than 10min (default value).
> latest git plugin allows us to set a different value than 10min.
>
> for 1st action, i'd change those failing jobs to use maybe 20min instead,
> and see if it's fixed.
> can you provide list of links for the failing jobs you saw?
>
> for longer run, i'd think on how to reduce/manage load on gerrit server,
> maybe using zuul. [1]
> david - any thoughts?
>
> [1] http://ci.openstack.org/zuul/
>
> ----- Original Message -----
>> From: "Michal Skrivanek" <michal.skrivanek(a)redhat.com>
>> To: infra(a)ovirt.org
>> Sent: Thursday, June 5, 2014 5:36:59 PM
>> Subject: Jenkins issue (findbugs)
>>
>>
>>> ----- Forwarded Message -----
>>> From: "Frantisek Kobzik" <fkobzik(a)redhat.com>
>>> To: infra(a)ovirt.org
>>> Sent: Thursday, June 5, 2014 11:23:12 AM
>>> Subject: Jenkins issue (findbugs)
>>>
>>> Hi guys,
>>>
>>> I have an issue with jenkins build and I was advised to contact infra.
>>> The console output is below (Similar issue also happens with other kinds
>>> of
>>> jobs).
>>>
>>> I'm seeing this issue quite often (~50 % of pushes), is there anything
>>> you
>>> can do about it?
>>>
>>> Thanks,
>>> Franta.
>>>
>>>
>>> 9:47:36 ERROR: Timeout after 10 minutes
>>> 09:47:36 FATAL: Failed to fetch from git://gerrit.ovirt.org/ovirt-engine
>>> 09:47:37 hudson.plugins.git.GitException: Failed to fetch from
>>> git://gerrit.ovirt.org/ovirt-engine
>>> 09:47:37 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:623)
>>> 09:47:37 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:855)
>>> 09:47:37 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:880)
>>> 09:47:37 at
>>> org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:118)
>>> 09:47:37 at
>>> hudson.model.AbstractProject.checkout(AbstractProject.java:1320)
>>> 09:47:37 at
>>>
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:609)
>>> 09:47:37 at
>>> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
>>> 09:47:37 at
>>>
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:518)
>>> 09:47:37 at hudson.model.Run.execute(Run.java:1700)
>>> 09:47:37 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>>> 09:47:37 at
>>> hudson.model.ResourceController.execute(ResourceController.java:88)
>>> 09:47:37 at hudson.model.Executor.run(Executor.java:231)
>>> 09:47:37 Caused by: hudson.plugins.git.GitException: Command "git
fetch
>>> --tags --progress git://gerrit.ovirt.org/ovirt-engine
>>> refs/changes/14/28314/3 --prune" returned status code 143:
>>> 09:47:37 stdout:
>>> 9:47:37 stderr: remote: Counting objects: 136
>>> remote: Counting objects: 375
>>> remote: Counting objects: 582
>>> .
>>> etc.
>>> .
>>> 09:47:37 at
>>>
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1276)
>>> 09:47:37 at
>>>
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1146)
>>> 09:47:37 at
>>>
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImpl.java:87)
>>> 09:47:37 at
>>>
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:254)
>>> 09:47:37 at
>>>
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
>>> 09:47:37 at
>>>
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
>>> 09:47:37 at hudson.remoting.UserRequest.perform(UserRequest.java:118)
>>> 09:47:37 at hudson.remoting.UserRequest.perform(UserRequest.java:48)
>>> 09:47:37 at hudson.remoting.Request$2.run(Request.java:328)
>>> 09:47:37 at
>>>
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
>>> 09:47:37 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> 09:47:37 at
>>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> 09:47:37 at
>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> 09:47:37 at java.lang.Thread.run(Thread.java:744)
>>> .
>>
>> _______________________________________________
>> Infra mailing list
>> Infra(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/infra
>>
Maybe we can start investigating on create mirrors for gerrit to pull
from the slaves (also downstream if able) at least for read operations,
that will leverage a lot of load from the jenkins slaves (if not all of
it)
--
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Email: dcaro(a)redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
3916
days inactive
3916
days old
3 comments
3 participants
participants (3)
-
David Caro -
Eyal Edri -
Michal Skrivanek