This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
------enig2XDNHUJGOXFAFUXTETMOM
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
One of the problems that I've seen is that xinetd is configured to a low
limit of simultaneous connections and it's dropping them when a few
jenkins jobs are running in parallel.
Changing/adding these options in the /etc/xinit.d/git file should be
enough (those are the actual values):
# Connection per second and time to wait when reached
cps =3D 50 10
# Max number of parallel instances
instances =3D 50
# Max number of connection per source
per_source =3D 10
Here is the trace of one of the failures:
Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit
from=3D::ffff:XX.XX.XX.X1
Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit
from=3D::ffff:XX.XX.XX.X2
Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit
from=3D::ffff:XX.XX.XX.X2
Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit
from=3D::ffff:XX.XX.XX.X2
Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit
from=3D::ffff:XX.XX.XX.X2
Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit
from=3D::ffff:XX.XX.XX.X2
Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit
from=3D::ffff:XX.XX.XX.X3
Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit
from=3D::ffff:XX.XX.XX.X4
Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit
from=3D::ffff:XX.XX.XX.X2
Jun 6 10:30:08 gerrit xinetd[1153]: FAIL: git service_limit
from=3D::ffff:XX.XX.XX.X5
Jun 6 10:30:08 gerrit xinetd[1153]: EXIT: git status=3D0 pid=3D340
duration=3D1(sec)
### The XX.XX.XX.XX are obfuscated ips
Also I've seen that the number of open files is stable and between
reasonable values, that's good :)
And memory and load seem fine too.
I'll keep monitoring it and see if I see something more, but I recommend
raising the values and see if it's enough of there's another underlying
problem.
--=20
David Caro
Red Hat Czech s.r.o.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605
Email: dcaro(a)redhat.com
Web:
www.cz.redhat.com
Red Hat Czech s.r.o., Purky=C5=88ova 99/71, 612 45, Brno, Czech Republic
RHT Global #: 82-62605
------enig2XDNHUJGOXFAFUXTETMOM
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (GNU/Linux)
iQEcBAEBAgAGBQJRsKBlAAoJEEBxx+HSYmnDeAoH/iUjpE7mk9LT1aA50/4IttXX
WI2Id5ZSCQEdZISftJ2sQAH1qusYKR9dU/O9+Ptm2n8Pw5/HO9ADWw0T7A1H1OJ7
4jMYg/xdGpOQIWjFhTGSuggu7xiWDaAc9+CiRJcpY6dIlVaGkghSAUn4/DCfdqCx
5aayKvZH9SQtAnO9DMCrJkvYPsFmLN6Cs8+sni4lov+KQabg6po+B7rDNV1AeeO0
cmBS7ckwEe1UZl8tvqPcEW+4Yr7LJkPROEOqf2p6KBBnPTXN6GJAa/bl+WFhotZz
IzskqnxM7AvKNoJF7fyVPGms6YlmeY+RphUfKh09/2xhsQhXWU4He6ke4BjJKps=
=XfuM
-----END PGP SIGNATURE-----
------enig2XDNHUJGOXFAFUXTETMOM--