
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --vemwgTXcXfVit5q3KcquwhO97dNC98WaD Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi! I think I've found the main cause for the latest gerrit load problems, we= must run gerrit gc periodically to tidy up the repos, I've ran it today on all= the repos and ovirt-engine passed from ~20 min to clone to ~2 minutes on my m= achine. I'll set up a cron to run it periodically, let me know if you see any iss= ues with it. FYI I've also changed some engine jobs to clone first from github and the= n fetch from gerrit, to ease the load and bandwidth on gerrit itself, I've tested= it on gerrit and non gerrit jobs before setting it up so no problem should come= from it, but don't get surprised if you see an extra git repo on the jobs conf= igs. Cheers! --=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --vemwgTXcXfVit5q3KcquwhO97dNC98WaD Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJTm2erAAoJEEBxx+HSYmnDxy8H/0kLpht8Sh3Z4wzxhpXIqBCb WOc3yhwLVJDNy5pcIGLYo9nB5gCuLY8T9+uB843HZ845scnCKtbPfQipgLrjY92a 4RfBhDjBPPNrgj1rEKbufNCrsAtS85URa8Z2t/POcUaTeqtnjx67LljJGTgTnc9M AOZ/I57pssNzVtO7hGV0vxPZZ42cv0HTVbqiiPidGH4btYqx/PWm85/BSig2K+0L x3pjb63IN3lHO7ZNaXhJ4VZV1fAK06ZOTbg7UjnfvlOnxArkFWCujrrBYmD+WX/Z pf1CjaHGCaH6Vu/CEY582rRyykH23gMGoLbEQtUoyt9Sa6p7v4tlttSQnMv0DTM= =BppM -----END PGP SIGNATURE----- --vemwgTXcXfVit5q3KcquwhO97dNC98WaD--

Il 13/06/2014 23:05, David Caro ha scritto:
Hi!
I think I've found the main cause for the latest gerrit load problems, we must run gerrit gc periodically to tidy up the repos, I've ran it today on all the repos and ovirt-engine passed from ~20 min to clone to ~2 minutes on my machine.
I'll set up a cron to run it periodically, let me know if you see any issues with it.
FYI I've also changed some engine jobs to clone first from github and then fetch from gerrit, to ease the load and bandwidth on gerrit itself, I've tested it on gerrit and non gerrit jobs before setting it up so no problem should come from it, but don't get surprised if you see an extra git repo on the jobs configs.
Looks like it's not working as expected: http://jenkins.ovirt.org/job/ovirt-engine_master_create-rpms-quick_gerrit/26... 09:50:12 Retriggered by user sbonazzo for Gerrit: http://gerrit.ovirt.org/28579 09:50:12 Building remotely on os1-rhel6-vm02 (centos6 os1) in workspace /home/jenkins/workspace/ovirt-engine_master_create-rpms-quick_gerrit 09:50:14 Fetching changes from the remote Git repository 09:50:23 Cleaning workspace 09:50:23 Resetting working tree 09:50:24 Fetching upstream changes from https://github.com/oVirt/ovirt-engine.git 09:50:26 Checking out Revision fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c (origin/master) 09:50:27 FATAL: Could not checkout null with start point fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c 09:50:27 hudson.plugins.git.GitException: Could not checkout null with start point fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c 09:50:27 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$8.execute(CliGitAPIImpl.java:1448) 09:50:27 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) 09:50:27 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) 09:50:27 at hudson.remoting.UserRequest.perform(UserRequest.java:118) 09:50:27 at hudson.remoting.UserRequest.perform(UserRequest.java:48) 09:50:27 at hudson.remoting.Request$2.run(Request.java:328) 09:50:27 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) 09:50:27 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 09:50:27 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 09:50:27 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 09:50:27 at java.lang.Thread.run(Thread.java:744) 09:50:27 Caused by: hudson.plugins.git.GitException: Command "git checkout -f fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c" returned status code 128: 09:50:27 stdout: 09:50:27 stderr: fatal: unable to write new index file
Cheers!
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

----- Original Message -----
From: "Sandro Bonazzola" <sbonazzo@redhat.com> To: "David Caro" <dcaroest@redhat.com>, "infra" <infra@ovirt.org> Sent: Saturday, June 14, 2014 11:52:15 AM Subject: Re: Gerrit load
Il 13/06/2014 23:05, David Caro ha scritto:
Hi!
I think I've found the main cause for the latest gerrit load problems, we must run gerrit gc periodically to tidy up the repos, I've ran it today on all the repos and ovirt-engine passed from ~20 min to clone to ~2 minutes on my machine.
I'll set up a cron to run it periodically, let me know if you see any issues with it.
FYI I've also changed some engine jobs to clone first from github and then fetch from gerrit, to ease the load and bandwidth on gerrit itself, I've tested it on gerrit and non gerrit jobs before setting it up so no problem should come from it, but don't get surprised if you see an extra git repo on the jobs configs.
Looks like it's not working as expected:
http://jenkins.ovirt.org/job/ovirt-engine_master_create-rpms-quick_gerrit/26...
seems to be working now, maybe a temp error? we should however monitor if github starts blocking connections due to load. and add another mirror we can use to offload (maybe similar to universities mirrors).
09:50:12 Retriggered by user sbonazzo for Gerrit: http://gerrit.ovirt.org/28579 09:50:12 Building remotely on os1-rhel6-vm02 (centos6 os1) in workspace /home/jenkins/workspace/ovirt-engine_master_create-rpms-quick_gerrit 09:50:14 Fetching changes from the remote Git repository 09:50:23 Cleaning workspace 09:50:23 Resetting working tree 09:50:24 Fetching upstream changes from https://github.com/oVirt/ovirt-engine.git 09:50:26 Checking out Revision fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c (origin/master) 09:50:27 FATAL: Could not checkout null with start point fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c 09:50:27 hudson.plugins.git.GitException: Could not checkout null with start point fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c 09:50:27 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$8.execute(CliGitAPIImpl.java:1448) 09:50:27 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) 09:50:27 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) 09:50:27 at hudson.remoting.UserRequest.perform(UserRequest.java:118) 09:50:27 at hudson.remoting.UserRequest.perform(UserRequest.java:48) 09:50:27 at hudson.remoting.Request$2.run(Request.java:328) 09:50:27 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) 09:50:27 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 09:50:27 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 09:50:27 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 09:50:27 at java.lang.Thread.run(Thread.java:744) 09:50:27 Caused by: hudson.plugins.git.GitException: Command "git checkout -f fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c" returned status code 128: 09:50:27 stdout: 09:50:27 stderr: fatal: unable to write new index file
Cheers!
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --SEpkd7bAS3j2Q17FXRUkmGMblhcRi7AcH Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Sun 15 Jun 2014 03:28:45 PM CEST, Eyal Edri wrote:
----- Original Message -----
From: "Sandro Bonazzola" <sbonazzo@redhat.com> To: "David Caro" <dcaroest@redhat.com>, "infra" <infra@ovirt.org> Sent: Saturday, June 14, 2014 11:52:15 AM Subject: Re: Gerrit load
Il 13/06/2014 23:05, David Caro ha scritto:
Hi!
I think I've found the main cause for the latest gerrit load problems=
, we
must run gerrit gc periodically to tidy up the repos, I've ran it today on= all the repos and ovirt-engine passed from ~20 min to clone to ~2 minutes on = my machine.
I'll set up a cron to run it periodically, let me know if you see any=
issues with it.
FYI I've also changed some engine jobs to clone first from github and= then fetch from gerrit, to ease the load and bandwidth on gerrit itself, I've te= sted it on gerrit and non gerrit jobs before setting it up so no problem should = come from it, but don't get surprised if you see an extra git repo on the jobs configs.
Looks like it's not working as expected:
http://jenkins.ovirt.org/job/ovirt-engine_master_create-rpms-quick_ger= rit/2603/console
seems to be working now, maybe a temp error? we should however monitor if github starts blocking connections due to = load. and add another mirror we can use to offload (maybe similar to universi= ties mirrors).
09:50:12 Retriggered by user sbonazzo for Gerrit: http://gerrit.ovirt.org/28579 09:50:12 Building remotely on os1-rhel6-vm02 (centos6 os1) in workspac= e /home/jenkins/workspace/ovirt-engine_master_create-rpms-quick_gerrit 09:50:14 Fetching changes from the remote Git repository 09:50:23 Cleaning workspace 09:50:23 Resetting working tree 09:50:24 Fetching upstream changes from https://github.com/oVirt/ovirt-engine.git 09:50:26 Checking out Revision fdf1b82f8cb6d643a7267535c4b2faeabfe94d1= c (origin/master) 09:50:27 FATAL: Could not checkout null with start point fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c 09:50:27 hudson.plugins.git.GitException: Could not checkout null with= start point fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c 09:50:27 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$8.execute(CliGitAPIImpl.= java:1448) 09:50:27 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler= $1.call(RemoteGitImpl.java:153) 09:50:27 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler= $1.call(RemoteGitImpl.java:146) 09:50:27 at hudson.remoting.UserRequest.perform(UserRequest.java:118)=
09:50:27 at hudson.remoting.UserRequest.perform(UserRequest.java:48) 09:50:27 at hudson.remoting.Request$2.run(Request.java:328) 09:50:27 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecuto= rService.java:72) 09:50:27 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 09:50:27 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j= ava:1145) 09:50:27 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.= java:615) 09:50:27 at java.lang.Thread.run(Thread.java:744) 09:50:27 Caused by: hudson.plugins.git.GitException: Command "git chec= kout -f fdf1b82f8cb6d643a7267535c4b2faeabfe94d1c" returned status code 128: 09:50:27 stdout: 09:50:27 stderr: fatal: unable to write new index file
Cheers!
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaborati= on. See how it works at redhat.com _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
With the garbage collecting it seems that it improved a lot, maybe it's=20 not necessary to do this right now, but work looking into though. -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --SEpkd7bAS3j2Q17FXRUkmGMblhcRi7AcH Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJTnqI/AAoJEEBxx+HSYmnDQFMH/RuOIVfiIFD13L3x1In4IKxe Pcd3hwe+uxTSJfxhR4hHGpFNCXmZEVkimjmlzN3nTJHS/NW4DTuyc9DW3n2JNG0M +fPIDsggPiGK/L653OAn0LrcDgPJMpEGBSfTo5Hy58p4YxNXx6ifkfzm4alOmj+A 9oAzp1aExlixEjEabTRkGpKPiHxv/HvTCRsdKzhkHhSLI+hlI87rLKBTxG7qAoxF LogmQDfd2CjQh4LdrE1f/NsPLvHugQc9FW+ZPHZmkTHwIO39KebMHG9I1BevN/1J IoaMw9oF5rfYaoGKiFSLWDnK81O0q10l8zC6sk7qq1eWEYIt12jvSMqA6Py4wCc= =wf1N -----END PGP SIGNATURE----- --SEpkd7bAS3j2Q17FXRUkmGMblhcRi7AcH--
participants (3)
-
David Caro
-
Eyal Edri
-
Sandro Bonazzola