Tal Nisan has submitted this change and it was merged. Change subject: core: Use persistent HTTP connections between engine and SSO ...................................................................... core: Use persistent HTTP connections between engine and SSO The SSO service and the engine authentication filters use HTTP to talk to each other. The implementation of this HTTP dialog is such that a new connection is created for each request. In production environments HTTPS is enabled by default, and that means that for each request new SSL socket and session are created and a new SSL handshake is performed. This is bad for performance, in general, but in certain situations is is also a potential trigger of engine crashes. For example, lets assume that the engine is running in a machine with 2 GiB of RAM and a heap size of 1 GiB, and consider a client that is continually sending authentication requests to the API, the following Python SDK script, for example: #!/usr/bin/python import sys from ovirtsdk.api import API from ovirtsdk.xml import params while True: # Connect to the API: api = API( url="https://engine40.local/ovirt-engine/api", username="admin@internal", password="redhat123", ca_file="/etc/pki/ovirt-engine/ca.pem", ) # Do something ... # Disconnect: api.disconnect() This script, alone, will trigger the creation of thousands of SSL sockets and sessions in the engine, and in the web server. But the SSL socket class is finalizable, and there is space enough in the heap, so those thousands of sockets, already closed, will still be in memory, in the finalizer queue. But those thousands of sockets also hold native resources, like socket buffers, which aren't acounted for in the heap. The result is that the Java virtual machine will consume much more memory than what you would expect, memory that isn't part of the heap. The result, in that 2 GiB machine, is that the out of memory killer of the kernel will trigger, and kill the engine, even if it isn't using all its heap space. This could be addressed with smarter handling of the SSL sockets, but that is well beyond the scope of our project. Alternatively we can try to reuse the HTTP connections, which should save sockets, SSL sessions, SSL handshakes and TCP connections. This patch tries to improve the use of connections, introducing a pool of HTTP connections, where connections are reused as much as possible. The effect is visible running the above Python SDK script and counting the number of SSL socketes that are created: # su -s /bin/sh ovirt # watch 'jmap -histo $(pidof ovirt-engine) | grep SSLSocketImpl' Without this patch the number of sockets is ever increasing, till there engine crashes or there is a garbage collection. In the 2 GiB environment it is in the order of thousands of instances. With the patch, the number is limited to a max of 20 sockets. In the 2 GiB environment it is usually 2 sockets. The patch also introduces two new configuration variables that can be used to adjust the size of the pools of HTTP connections: # The maximum size of the pool of HTTP connections that # the engine uses to communicate with the SSO service: ENGINE_SSO_SERVICE_CLIENT_POOL_SIZE=10 # The maximum size of the pool of HTTP connections that # the SSO service uses to communicate with the engine: SSO_CALLBACK_CLIENT_POOL_SIZE=10 Change-Id: Ifa686b9f73c693ec20e0e51f2c004b6eea9e21bc Related-To: https://bugzilla.redhat.com/1396833 Signed-off-by: Juan Hernandez <juan.hernandez(a)redhat.com&gt; Signed-off-by: Ravi Nori <rnori(a)redhat.com&gt; (cherry picked from commit 88abb6e0f90858e422d249a7ccda7b6c5027aee5) Signed-off-by: Martin Perina <mperina(a)redhat.com&gt; --- M backend/manager/modules/aaa/pom.xml M backend/manager/modules/aaa/src/main/java/org/ovirt/engine/core/aaa/SsoOAuthServiceUtils.java M backend/manager/modules/aaa/src/main/java/org/ovirt/engine/core/aaa/filters/FiltersHelper.java M backend/manager/modules/aaa/src/main/modules/org/ovirt/engine/core/aaa/main/module.xml M backend/manager/modules/enginesso/pom.xml M backend/manager/modules/enginesso/src/main/java/org/ovirt/engine/core/sso/servlets/InteractiveChangePasswdServlet.java M backend/manager/modules/enginesso/src/main/java/org/ovirt/engine/core/sso/utils/SsoUtils.java M backend/manager/modules/uutils/pom.xml A backend/manager/modules/uutils/src/main/java/org/ovirt/engine/core/uutils/net/HttpClientBuilder.java M backend/manager/modules/uutils/src/main/modules/org/ovirt/engine/core/uutils/main/module.xml M ear/src/main/application/META-INF/jboss-deployment-structure.xml M packaging/services/ovirt-engine/ovirt-engine.conf.in 12 files changed, 437 insertions(+), 224 deletions(-) Approvals: Martin Peřina: Verified; Looks good to me, approved Jenkins CI: Passed CI tests -- To view, visit https://gerrit.ovirt.org/68394 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ifa686b9f73c693ec20e0e51f2c004b6eea9e21bc Gerrit-PatchSet: 3 Gerrit-Project: ovirt-engine Gerrit-Branch: ovirt-engine-4.0 Gerrit-Owner: Juan Hernandez <juan.hernandez(a)redhat.com&gt; Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Peřina <mperina(a)redhat.com&gt; Gerrit-Reviewer: Piotr Kliczewski <piotr.kliczewski(a)gmail.com&gt; Gerrit-Reviewer: Ravi Nori <rnori(a)redhat.com&gt; Gerrit-Reviewer: Tal Nisan <tnisan(a)redhat.com&gt; Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org&gt;