Change in ovirt-engine[ovirt-engine-4.0]: core: Use persistent HTTP connections between engine and SSO

[Code Review]

>From Tal Nisan <tnisan at redhat.com>:

Tal Nisan has submitted this change and it was merged.

Change subject: core: Use persistent HTTP connections between engine and SSO
......................................................................


core: Use persistent HTTP connections between engine and SSO

The SSO service and the engine authentication filters use HTTP
to talk to each other. The implementation of this HTTP dialog
is such that a new connection is created for each request. In
production environments HTTPS is enabled by default, and that
means that for each request new SSL socket and session are
created and a new SSL handshake is performed. This is bad for
performance, in general, but in certain situations is is also
a potential trigger of engine crashes. For example, lets
assume that the engine is running in a machine with 2 GiB of
RAM and a heap size of 1 GiB, and consider a client that is
continually sending authentication requests to the API, the
following Python SDK script, for example:

  #!/usr/bin/python

  import sys
  from ovirtsdk.api import API
  from ovirtsdk.xml import params

  while True:
    # Connect to the API:
    api = API(
      url="https://engine40.local/ovirt-engine/api",
      username="admin at internal",
      password="redhat123",
      ca_file="/etc/pki/ovirt-engine/ca.pem",
    )

    # Do something ...

    # Disconnect:
    api.disconnect()

This script, alone, will trigger the creation of thousands of
SSL sockets and sessions in the engine, and in the web server.
But the SSL socket class is finalizable, and there is space
enough in the heap, so those thousands of sockets, already
closed, will still be in memory, in the finalizer queue. But
those thousands of sockets also hold native resources, like
socket buffers, which aren't acounted for in the heap. The
result is that the Java virtual machine will consume much more
memory than what you would expect, memory that isn't part of
the heap. The result, in that 2 GiB machine, is that the out
of memory killer of the kernel will trigger, and kill the
engine, even if it isn't using all its heap space.

This could be addressed with smarter handling of the SSL
sockets, but that is well beyond the scope of our project.

Alternatively we can try to reuse the HTTP connections, which
should save sockets, SSL sessions, SSL handshakes and TCP
connections.

This patch tries to improve the use of connections,
introducing a pool of HTTP connections, where connections are
reused as much as possible.

The effect is visible running the above Python SDK script and
counting the number of SSL socketes that are created:

  # su -s /bin/sh ovirt
  # watch 'jmap -histo $(pidof ovirt-engine) | grep SSLSocketImpl'

Without this patch the number of sockets is ever increasing,
till there engine crashes or there is a garbage collection.
In the 2 GiB environment it is in the order of thousands of
instances.

With the patch, the number is limited to a max of 20 sockets.
In the 2 GiB environment it is usually 2 sockets.

The patch also introduces two new configuration variables that can be
used to adjust the size of the pools of HTTP connections:

  # The maximum size of the pool of HTTP connections that
  # the engine uses to communicate with the SSO service:
  ENGINE_SSO_SERVICE_CLIENT_POOL_SIZE=10

  # The maximum size of the pool of HTTP connections that
  # the SSO service uses to communicate with the engine:
  SSO_CALLBACK_CLIENT_POOL_SIZE=10

Change-Id: Ifa686b9f73c693ec20e0e51f2c004b6eea9e21bc
Related-To: https://bugzilla.redhat.com/1396833
Signed-off-by: Juan Hernandez <juan.hernandez at redhat.com>
Signed-off-by: Ravi Nori <rnori at redhat.com>
(cherry picked from commit 88abb6e0f90858e422d249a7ccda7b6c5027aee5)
Signed-off-by: Martin Perina <mperina at redhat.com>
---
M backend/manager/modules/aaa/pom.xml
M backend/manager/modules/aaa/src/main/java/org/ovirt/engine/core/aaa/SsoOAuthServiceUtils.java
M backend/manager/modules/aaa/src/main/java/org/ovirt/engine/core/aaa/filters/FiltersHelper.java
M backend/manager/modules/aaa/src/main/modules/org/ovirt/engine/core/aaa/main/module.xml
M backend/manager/modules/enginesso/pom.xml
M backend/manager/modules/enginesso/src/main/java/org/ovirt/engine/core/sso/servlets/InteractiveChangePasswdServlet.java
M backend/manager/modules/enginesso/src/main/java/org/ovirt/engine/core/sso/utils/SsoUtils.java
M backend/manager/modules/uutils/pom.xml
A backend/manager/modules/uutils/src/main/java/org/ovirt/engine/core/uutils/net/HttpClientBuilder.java
M backend/manager/modules/uutils/src/main/modules/org/ovirt/engine/core/uutils/main/module.xml
M ear/src/main/application/META-INF/jboss-deployment-structure.xml
M packaging/services/ovirt-engine/ovirt-engine.conf.in
12 files changed, 437 insertions(+), 224 deletions(-)

Approvals:
  Martin Peřina: Verified; Looks good to me, approved
  Jenkins CI: Passed CI tests



-- 
To view, visit https://gerrit.ovirt.org/68394
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ifa686b9f73c693ec20e0e51f2c004b6eea9e21bc
Gerrit-PatchSet: 3
Gerrit-Project: ovirt-engine
Gerrit-Branch: ovirt-engine-4.0
Gerrit-Owner: Juan Hernandez <juan.hernandez at redhat.com>
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Martin Peřina <mperina at redhat.com>
Gerrit-Reviewer: Piotr Kliczewski <piotr.kliczewski at gmail.com>
Gerrit-Reviewer: Ravi Nori <rnori at redhat.com>
Gerrit-Reviewer: Tal Nisan <tnisan at redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation at ovirt.org>