This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --o6RP2RcQD2tNiSTweTm6P1TSOuWK8O5Vf Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable El mi=C3=A9 15 ene 2014 21:06:41 CET, R P Herrold escribi=C3=B3:
On Wed, 15 Jan 2014, R P Herrold wrote:
Nagios reports it disappeared at 13:28:44 and re-appeared at 13:48:34
Hi, David
I see you said in IRC
13:42 < dcaro> ecohen: apuimedo|away yes, gerrit web is having issues, it started too many threads
What jobs were then running that you saw this? Did you deduce this from the process table, the dmesg, or some log file?
-- Russ herrold
Hi Russ, Well, more or less, I saw that the process count for gerrit was a=20 number really close to 1024, and that it was not increasing or=20 decreasing. Then I looked at the logs and sow a coupple of entries like=20 this: [2014-01-15 13:21:50,584] ERROR com.google.gerrit.pgm.Daemon : Thread=20 HTTP-23 threw exception java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:657) at=20 org.eclipse.jetty.util.thread.QueuedThreadPool.startThread(QueuedThreadPo= ol.java:441) at=20 org.eclipse.jetty.util.thread.QueuedThreadPool.dispatch(QueuedThreadPool.= java:366) at=20 org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorMana= ger.dispatch(SelectChannelConnector.java:300) at=20 org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManag= er.java:708) at=20 org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at=20 org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.ja= va:608) at=20 org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.jav= a:543) at java.lang.Thread.run(Thread.java:679) So yes, that seems to be the problem. I raised the limit for user=20 processes from 1024 to 10240 for now, but I have to check what is=20 making those processes get stuck. I'm not sure of the jobs that were running at that moment. But there=20 were a few and all get stuck trying to get the changes through http=20 protocol. -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --o6RP2RcQD2tNiSTweTm6P1TSOuWK8O5Vf Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJS1xx3AAoJEEBxx+HSYmnDuqgIAJLGOMhx2c2PZTWAe/OUwO6K r2+2xlgazsfzXhRSAckUYw/WBer6d57QMeHcbHbUBsY+xgxFO7jraJKIkvXUx8Ur zEMCSnL0jZwFoR57L/bll6onBaTSeMKFxgh9JvK9aFLKqlLJZSePcHnOtOjQDSeL vIMOXUjSg88fyP9SCLgarWLsl0EYEEPds2fCC554wN7CnOcPqS0hr/bxBHqRkeDJ UNPhmV0L3iODduiWKUXiS5A7Lcj1nrfpuL0KAkaVbS9aij/XWGi5o3zeB2puJqv+ FF1o2fnrwY1Mewlx83pJ7QS97cyT9krMQU5nFvX92xlo8cvINuDNPgXzsnCa3iU= =wmK/ -----END PGP SIGNATURE----- --o6RP2RcQD2tNiSTweTm6P1TSOuWK8O5Vf--