11:05 p.m.
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--vemwgTXcXfVit5q3KcquwhO97dNC98WaD
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Hi!
I think I've found the main cause for the latest gerrit load problems, we=
must
run gerrit gc periodically to tidy up the repos, I've ran it today on all=
the
repos and ovirt-engine passed from ~20 min to clone to ~2 minutes on my m=
achine.
I'll set up a cron to run it periodically, let me know if you see any iss=
ues
with it.
FYI I've also changed some engine jobs to clone first from github and the=
n fetch
from gerrit, to ease the load and bandwidth on gerrit itself, I've tested=
it on
gerrit and non gerrit jobs before setting it up so no problem should come=
from
it, but don't get surprised if you see an extra git repo on the jobs conf=
igs.
Cheers!
--=20
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Email: dcaro(a)redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
--vemwgTXcXfVit5q3KcquwhO97dNC98WaD
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJTm2erAAoJEEBxx+HSYmnDxy8H/0kLpht8Sh3Z4wzxhpXIqBCb
WOc3yhwLVJDNy5pcIGLYo9nB5gCuLY8T9+uB843HZ845scnCKtbPfQipgLrjY92a
4RfBhDjBPPNrgj1rEKbufNCrsAtS85URa8Z2t/POcUaTeqtnjx67LljJGTgTnc9M
AOZ/I57pssNzVtO7hGV0vxPZZ42cv0HTVbqiiPidGH4btYqx/PWm85/BSig2K+0L
x3pjb63IN3lHO7ZNaXhJ4VZV1fAK06ZOTbg7UjnfvlOnxArkFWCujrrBYmD+WX/Z
pf1CjaHGCaH6Vu/CEY582rRyykH23gMGoLbEQtUoyt9Sa6p7v4tlttSQnMv0DTM=
=BppM
-----END PGP SIGNATURE-----
--vemwgTXcXfVit5q3KcquwhO97dNC98WaD--
10:52 a.m.
Il 13/06/2014 23:05, David Caro ha scritto:
Hi!
I think I've found the main cause for the latest gerrit load problems, we must
run gerrit gc periodically to tidy up the repos, I've ran it today on all the
repos and ovirt-engine passed from ~20 min to clone to ~2 minutes on my machine.
I'll set up a cron to run it periodically, let me know if you see any issues
with it.
FYI I've also changed some engine jobs to clone first from github and then fetch
from gerrit, to ease the load and bandwidth on gerrit itself, I've tested it on
gerrit and non gerrit jobs before setting it up so no problem should come from
it, but don't get surprised if you see an extra git repo on the jobs configs.
Looks like it's not working as expected:
http://jenkins.ovirt.org/job/ovirt-engine_master_create-rpms-quick_gerrit...
09:50:12 Retriggered by user sbonazzo for Gerrit: http://gerrit.ovirt.org/28579
09:50:12 Building remotely on os1-rhel6-vm02 (centos6 os1) in workspace
/home/jenkins/workspace/ovirt-engine_master_create-rpms-quick_gerrit
09:50:14 Fetching changes from the remote Git repository
09:50:23 Cleaning workspace
09:50:23 Resetting working tree
09:50:24 Fetching upstream changes from https://github.com/oVirt/ovirt-engine.git
09:50:26 Checking out Revision fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c (origin/master)
09:50:27 FATAL: Could not checkout null with start point
fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c
09:50:27 hudson.plugins.git.GitException: Could not checkout null with start point
fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c
09:50:27 at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$8.execute(CliGitAPIImpl.java:1448)
09:50:27 at
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
09:50:27 at
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
09:50:27 at hudson.remoting.UserRequest.perform(UserRequest.java:118)
09:50:27 at hudson.remoting.UserRequest.perform(UserRequest.java:48)
09:50:27 at hudson.remoting.Request$2.run(Request.java:328)
09:50:27 at
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
09:50:27 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
09:50:27 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
09:50:27 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
09:50:27 at java.lang.Thread.run(Thread.java:744)
09:50:27 Caused by: hudson.plugins.git.GitException: Command "git checkout -f
fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c" returned status code 128:
09:50:27 stdout:
09:50:27 stderr: fatal: unable to write new index file
Cheers!
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
--
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com
3:28 p.m.
----- Original Message -----
From: "Sandro Bonazzola" <sbonazzo(a)redhat.com>
To: "David Caro" <dcaroest(a)redhat.com>, "infra"
<infra(a)ovirt.org>
Sent: Saturday, June 14, 2014 11:52:15 AM
Subject: Re: Gerrit load
Il 13/06/2014 23:05, David Caro ha scritto:
> Hi!
>
>
> I think I've found the main cause for the latest gerrit load problems, we
> must
> run gerrit gc periodically to tidy up the repos, I've ran it today on all
> the
> repos and ovirt-engine passed from ~20 min to clone to ~2 minutes on my
> machine.
>
> I'll set up a cron to run it periodically, let me know if you see any
> issues
> with it.
>
> FYI I've also changed some engine jobs to clone first from github and then
> fetch
> from gerrit, to ease the load and bandwidth on gerrit itself, I've tested
> it on
> gerrit and non gerrit jobs before setting it up so no problem should come
> from
> it, but don't get surprised if you see an extra git repo on the jobs
> configs.
Looks like it's not working as expected:
http://jenkins.ovirt.org/job/ovirt-engine_master_create-rpms-quick_gerrit...
seems to be working now, maybe a temp error?
we should however monitor if github starts blocking connections due to load.
and add another mirror we can use to offload (maybe similar to universities mirrors).
09:50:12 Retriggered by user sbonazzo for Gerrit:
http://gerrit.ovirt.org/28579
09:50:12 Building remotely on os1-rhel6-vm02 (centos6 os1) in workspace
/home/jenkins/workspace/ovirt-engine_master_create-rpms-quick_gerrit
09:50:14 Fetching changes from the remote Git repository
09:50:23 Cleaning workspace
09:50:23 Resetting working tree
09:50:24 Fetching upstream changes from
https://github.com/oVirt/ovirt-engine.git
09:50:26 Checking out Revision fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c
(origin/master)
09:50:27 FATAL: Could not checkout null with start point
fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c
09:50:27 hudson.plugins.git.GitException: Could not checkout null with start
point fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c
09:50:27 at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$8.execute(CliGitAPIImpl.java:1448)
09:50:27 at
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
09:50:27 at
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
09:50:27 at hudson.remoting.UserRequest.perform(UserRequest.java:118)
09:50:27 at hudson.remoting.UserRequest.perform(UserRequest.java:48)
09:50:27 at hudson.remoting.Request$2.run(Request.java:328)
09:50:27 at
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
09:50:27 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
09:50:27 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
09:50:27 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
09:50:27 at java.lang.Thread.run(Thread.java:744)
09:50:27 Caused by: hudson.plugins.git.GitException: Command "git checkout -f
fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c" returned status code 128:
09:50:27 stdout:
09:50:27 stderr: fatal: unable to write new index file
>
>
> Cheers!
>
>
>
> _______________________________________________
> Infra mailing list
> Infra(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>
--
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
9:52 a.m.
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--SEpkd7bAS3j2Q17FXRUkmGMblhcRi7AcH
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
On Sun 15 Jun 2014 03:28:45 PM CEST, Eyal Edri wrote:
----- Original Message -----
> From: "Sandro Bonazzola" <sbonazzo(a)redhat.com>
> To: "David Caro" <dcaroest(a)redhat.com>, "infra"
<infra(a)ovirt.org>
> Sent: Saturday, June 14, 2014 11:52:15 AM
> Subject: Re: Gerrit load
>
> Il 13/06/2014 23:05, David Caro ha scritto:
>> Hi!
>>
>>
>> I think I've found the main cause for the latest gerrit load problems=
, we
>> must
>> run gerrit gc periodically to tidy up the repos, I've ran it today on=
all
>> the
>> repos and ovirt-engine passed from ~20 min to clone to ~2 minutes on =
my
>> machine.
>>
>> I'll set up a cron to run it periodically, let me know if you see any=
>> issues
>> with it.
>>
>> FYI I've also changed some engine jobs to clone first from github and=
then
>> fetch
>> from gerrit, to ease the load and bandwidth on gerrit itself, I've te=
sted
>> it on
>> gerrit and non gerrit jobs before setting it up so no problem should =
come
>> from
>> it, but don't get surprised if you see an extra git repo on the jobs
>> configs.
>
>
> Looks like it's not working as expected:
>
> http://jenkins.ovirt.org/job/ovirt-engine_master_create-rpms-quick_ger=
rit/2603/console
>
seems to be working now, maybe a temp error?
we should however monitor if github starts blocking connections due to =
load.
and add another mirror we can use to offload (maybe similar to
universi=
ties mirrors).
> 09:50:12 Retriggered by user sbonazzo for Gerrit:
> http://gerrit.ovirt.org/28579
> 09:50:12 Building remotely on os1-rhel6-vm02 (centos6 os1) in workspac=
e
>
/home/jenkins/workspace/ovirt-engine_master_create-rpms-quick_gerrit
> 09:50:14 Fetching changes from the remote Git repository
> 09:50:23 Cleaning workspace
> 09:50:23 Resetting working tree
> 09:50:24 Fetching upstream changes from
> https://github.com/oVirt/ovirt-engine.git
> 09:50:26 Checking out Revision fdf1b82f8cb6d643a7267535c4b2faeabfe94d1=
c
> (origin/master)
> 09:50:27 FATAL: Could not checkout null with start point
> fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c
> 09:50:27 hudson.plugins.git.GitException: Could not checkout null with=
start
> point fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c
> 09:50:27 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$8.execute(CliGitAPIImpl.=
java:1448)
> 09:50:27 at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler=
$1.call(RemoteGitImpl.java:153)
> 09:50:27 at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler=
$1.call(RemoteGitImpl.java:146)
> 09:50:27 at
hudson.remoting.UserRequest.perform(UserRequest.java:118)=
> 09:50:27 at
hudson.remoting.UserRequest.perform(UserRequest.java:48)
> 09:50:27 at hudson.remoting.Request$2.run(Request.java:328)
> 09:50:27 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecuto=
rService.java:72)
> 09:50:27 at
java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 09:50:27 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j=
ava:1145)
> 09:50:27 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.=
java:615)
> 09:50:27 at java.lang.Thread.run(Thread.java:744)
> 09:50:27 Caused by: hudson.plugins.git.GitException: Command "git chec=
kout -f
> fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c" returned status
code 128:
> 09:50:27 stdout:
> 09:50:27 stderr: fatal: unable to write new index file
>
>
>>
>>
>> Cheers!
>>
>>
>>
>> _______________________________________________
>> Infra mailing list
>> Infra(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/infra
>>
>
>
> --
> Sandro Bonazzola
> Better technology. Faster innovation. Powered by community collaborati=
on.
> See how it works at redhat.com
> _______________________________________________
> Infra mailing list
> Infra(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>
With the garbage collecting it seems that it improved a lot, maybe it's=20
not necessary to do this right now, but work looking into though.
--
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Email: dcaro(a)redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
--SEpkd7bAS3j2Q17FXRUkmGMblhcRi7AcH
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJTnqI/AAoJEEBxx+HSYmnDQFMH/RuOIVfiIFD13L3x1In4IKxe
Pcd3hwe+uxTSJfxhR4hHGpFNCXmZEVkimjmlzN3nTJHS/NW4DTuyc9DW3n2JNG0M
+fPIDsggPiGK/L653OAn0LrcDgPJMpEGBSfTo5Hy58p4YxNXx6ifkfzm4alOmj+A
9oAzp1aExlixEjEabTRkGpKPiHxv/HvTCRsdKzhkHhSLI+hlI87rLKBTxG7qAoxF
LogmQDfd2CjQh4LdrE1f/NsPLvHugQc9FW+ZPHZmkTHwIO39KebMHG9I1BevN/1J
IoaMw9oF5rfYaoGKiFSLWDnK81O0q10l8zC6sk7qq1eWEYIt12jvSMqA6Py4wCc=
=wf1N
-----END PGP SIGNATURE-----
--SEpkd7bAS3j2Q17FXRUkmGMblhcRi7AcH--
3866
days inactive
3869
days old
3 comments
3 participants
participants (3)
-
David Caro -
Eyal Edri -
Sandro Bonazzola