[ https://ovirt-jira.atlassian.net/browse/OVIRT-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=38469#comment-38469 ] Evgheni Dereveanchin commented on OVIRT-2586: --------------------------------------------- I planned to reboot Jenkins tonight but it was busy running pipelines that aren't easy to cancel. As a result, Jenkins completely locked up in the morning with the UI being completely unreachable and backend threads timing out in the background. Had to restart it and it's now coming back up. The monitoring plugin was partially responsive still during the outage and showed the following info: |Java memory used: |15,590 Mb / 16,384 Mb *Usage is near the maximum, you may need to optimize or to reconfigure (-Xmx)| |Nb of http sessions: |8 | |Nb of active threads (current http requests): |33 | |System load |2.78 | |% System CPU |17.17| Almost all memory got exhausted which is likely caused by a memory leak in the SSE-gateway plugin coinciding with a large number of CI jobs appearing in the queue. Adding memory to Java will likely just delay the symptoms as the memory leak is still there (see JENKINS-51057) !usedMemory_year.png|thumbnail! From the yearly memory graph the leak started around May-June this year and intensified in November. To confirm the exact root cause we may need some lower-level troubleshooting of the Java process yet I am not familiar with how that's done. [~mwperina] maybe you can assist with the info that can be gathered to identify the root cause?

Jenkins terribly slow and unresponsive --------------------------------------

Key: OVIRT-2586 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2586 Project: oVirt - virtualization made easy Issue Type: Outage Reporter: sbonazzo Assignee: Evgheni Dereveanchin Priority: Highest Attachments: usedMemory_year.png

Hi, jenkins is terribly slow and becoming worse every day. I tried to gain some speed by adding 4 cores to the VM through engine-phx. It's a bit better but the real issue doesn't seem related to CPU power. Can anybody investigate? -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://red.ht/sig>

-- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)