gerrit.ovirt.org down?

[David Caro]

El mié 15 ene 2014 21:06:41 CET, R P Herrold escribió:
> On Wed, 15 Jan 2014, R P Herrold wrote:
>
>> Nagios reports it disappeared at 13:28:44
>> and re-appeared at 13:48:34
>
> Hi, David
>
> I see you said in IRC
>
> 13:42 < dcaro> ecohen: apuimedo|away yes, gerrit web is having
> 	issues, it started too many threads
>
> What jobs were then running that you saw this? Did you deduce
> this from the process table, the dmesg, or some log file?
>
> -- Russ herrold

Hi Russ,

Well, more or less, I saw that the process count for gerrit was a 
number really close to 1024, and that it was not increasing or 
decreasing. Then I looked at the logs and sow a coupple of entries like 
this:

[2014-01-15 13:21:50,584] ERROR com.google.gerrit.pgm.Daemon : Thread 
HTTP-23 threw exception
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:657)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.startThread(QueuedThreadPool.java:441)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.dispatch(QueuedThreadPool.java:366)
        at 
org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager.dispatch(SelectChannelConnector.java:300)
        at 
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:708)
        at 
org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)

So yes, that seems to be the problem. I raised the limit for user 
processes from 1024 to 10240 for now, but I have to check what is 
making those processes get stuck.


I'm not sure of the jobs that were running at that moment. But there 
were a few and all get stuck trying to get the changes through http 
protocol.

--
David Caro

Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D

Email: dcaro at redhat.com
Web: www.redhat.com
RHT Global #: 82-62605

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20140116/d0d872ed/attachment.sig>