
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) ------enig2XDNHUJGOXFAFUXTETMOM Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable One of the problems that I've seen is that xinetd is configured to a low limit of simultaneous connections and it's dropping them when a few jenkins jobs are running in parallel. Changing/adding these options in the /etc/xinit.d/git file should be enough (those are the actual values): # Connection per second and time to wait when reached cps =3D 50 10 # Max number of parallel instances instances =3D 50 # Max number of connection per source per_source =3D 10 Here is the trace of one of the failures: Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit from=3D::ffff:XX.XX.XX.X1 Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit from=3D::ffff:XX.XX.XX.X2 Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit from=3D::ffff:XX.XX.XX.X2 Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit from=3D::ffff:XX.XX.XX.X2 Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit from=3D::ffff:XX.XX.XX.X2 Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit from=3D::ffff:XX.XX.XX.X2 Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit from=3D::ffff:XX.XX.XX.X3 Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit from=3D::ffff:XX.XX.XX.X4 Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit from=3D::ffff:XX.XX.XX.X2 Jun 6 10:30:08 gerrit xinetd[1153]: FAIL: git service_limit from=3D::ffff:XX.XX.XX.X5 Jun 6 10:30:08 gerrit xinetd[1153]: EXIT: git status=3D0 pid=3D340 duration=3D1(sec) ### The XX.XX.XX.XX are obfuscated ips Also I've seen that the number of open files is stable and between reasonable values, that's good :) And memory and load seem fine too. I'll keep monitoring it and see if I see something more, but I recommend raising the values and see if it's enough of there's another underlying problem. --=20 David Caro Red Hat Czech s.r.o. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.cz.redhat.com Red Hat Czech s.r.o., Purky=C5=88ova 99/71, 612 45, Brno, Czech Republic RHT Global #: 82-62605 ------enig2XDNHUJGOXFAFUXTETMOM Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) iQEcBAEBAgAGBQJRsKBlAAoJEEBxx+HSYmnDeAoH/iUjpE7mk9LT1aA50/4IttXX WI2Id5ZSCQEdZISftJ2sQAH1qusYKR9dU/O9+Ptm2n8Pw5/HO9ADWw0T7A1H1OJ7 4jMYg/xdGpOQIWjFhTGSuggu7xiWDaAc9+CiRJcpY6dIlVaGkghSAUn4/DCfdqCx 5aayKvZH9SQtAnO9DMCrJkvYPsFmLN6Cs8+sni4lov+KQabg6po+B7rDNV1AeeO0 cmBS7ckwEe1UZl8tvqPcEW+4Yr7LJkPROEOqf2p6KBBnPTXN6GJAa/bl+WFhotZz IzskqnxM7AvKNoJF7fyVPGms6YlmeY+RphUfKh09/2xhsQhXWU4He6ke4BjJKps= =XfuM -----END PGP SIGNATURE----- ------enig2XDNHUJGOXFAFUXTETMOM--

On 06/06/2013 05:44 PM, David Caro wrote:
One of the problems that I've seen is that xinetd is configured to a low limit of simultaneous connections and it's dropping them when a few jenkins jobs are running in parallel.
Changing/adding these options in the /etc/xinit.d/git file should be enough (those are the actual values): # Connection per second and time to wait when reached cps = 50 10 # Max number of parallel instances instances = 50 # Max number of connection per source per_source = 10
Here is the trace of one of the failures:
Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit from=::ffff:XX.XX.XX.X1 Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit from=::ffff:XX.XX.XX.X2 Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit from=::ffff:XX.XX.XX.X2 Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit from=::ffff:XX.XX.XX.X2 Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit from=::ffff:XX.XX.XX.X2 Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit from=::ffff:XX.XX.XX.X2 Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit from=::ffff:XX.XX.XX.X3 Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit from=::ffff:XX.XX.XX.X4 Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit from=::ffff:XX.XX.XX.X2 Jun 6 10:30:08 gerrit xinetd[1153]: FAIL: git service_limit from=::ffff:XX.XX.XX.X5 Jun 6 10:30:08 gerrit xinetd[1153]: EXIT: git status=0 pid=340 duration=1(sec)
### The XX.XX.XX.XX are obfuscated ips
Also I've seen that the number of open files is stable and between reasonable values, that's good :) And memory and load seem fine too.
I'll keep monitoring it and see if I see something more, but I recommend raising the values and see if it's enough of there's another underlying problem.
thanks, please go ahead and do the xinit.d change.
participants (2)
-
David Caro
-
Itamar Heim