On Wed, 15 Jan 2014, Einav Cohen wrote: Nagios reports it disappeared at 13:28:44 and re-appeared at 13:48:34 US EST (GMT - 0500) -- Russ herrold

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --o6RP2RcQD2tNiSTweTm6P1TSOuWK8O5Vf Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable El mi=C3=A9 15 ene 2014 21:06:41 CET, R P Herrold escribi=C3=B3: Hi Russ, Well, more or less, I saw that the process count for gerrit was a=20 number really close to 1024, and that it was not increasing or=20 decreasing. Then I looked at the logs and sow a coupple of entries like=20 this: [2014-01-15 13:21:50,584] ERROR com.google.gerrit.pgm.Daemon : Thread=20 HTTP-23 threw exception java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:657) at=20 org.eclipse.jetty.util.thread.QueuedThreadPool.startThread(QueuedThreadPo= ol.java:441) at=20 org.eclipse.jetty.util.thread.QueuedThreadPool.dispatch(QueuedThreadPool.= java:366) at=20 org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorMana= ger.dispatch(SelectChannelConnector.java:300) at=20 org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManag= er.java:708) at=20 org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at=20 org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.ja= va:608) at=20 org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.jav= a:543) at java.lang.Thread.run(Thread.java:679) So yes, that seems to be the problem. I raised the limit for user=20 processes from 1024 to 10240 for now, but I have to check what is=20 making those processes get stuck. I'm not sure of the jobs that were running at that moment. But there=20 were a few and all get stuck trying to get the changes through http=20 protocol. -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Email: dcaro(a)redhat.com Web: www.redhat.com RHT Global #: 82-62605 --o6RP2RcQD2tNiSTweTm6P1TSOuWK8O5Vf Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" --o6RP2RcQD2tNiSTweTm6P1TSOuWK8O5Vf--