Hi Juan,

Now we got a clue, I try to hit 100 concurrent request in which 1 request got failed with the 503 error, I observed this in apache ssl_access.log

172.30.36.167 - - [08/Mar/2018:17:22:42 +0530] "GET /ovirt-engine/api/vms/ HTTP/1.1" 503 299

Is it thrown from Apache / ovirt engine?

On Thu, Mar 8, 2018 at 5:15 PM, Juan Hernández <jhernand@redhat.com> wrote:

I think the configuration is good. There may be some connections that the server is closing before the client expected, but that is normal, in my opinion.

On 03/08/2018 12:34 PM, Hari Prasanth Loganathan wrote:

This is the only error message we received from ab.

I googled it and found that it is due to the connection drop. So It would
be Great, If you could check my Apache server configuration I shared in the
thread and let me know your thoughts on this.

Thanks,
Hari

On Thu, Mar 8, 2018 at 4:56 PM, Juan Hernández <jhernand@redhat.com> wrote:

But other than those SSL error messages, are the connections really
failing? Can you share the results reported by "ab"?


On 03/08/2018 12:16 PM, Hari Prasanth Loganathan wrote:

No Juan, It is not working with any benchmark / application tool. It fails
with the same error SSL handshake failed (5).

Could you let me know the configuration of Apache web server is correct?

Thanks,
Hari

On Thu, Mar 8, 2018 at 1:08 AM, Juan Hernández <jhernand@redhat.com>
wrote:

If you are still having problems I am inclined to think that it is a

client issue. For example, I'd try to remove the "-k" option from the
"ab"
command. If you use keep alive the server may decide anyhow to close the
connection after certain number of requests, even if the client asks to
keep it alive. Some clients don't handle that perfectly, "ab" may have
that
problem. If that makes the SSL error messages disappear then I think you
can safely ignore them, and restore the "-k" option, if you want.

On 03/07/2018 07:30 PM, Hari Prasanth Loganathan wrote:

Thanks Juan for your response. Appreciate it.

But for some reason still, I am facing the same SSL handshake failed
(5).
Could you please check this configuration and let me know the issue in
my
ovirt engine environment.

*Configuration of Apache server:*


1) httpd version,

# httpd -v
Server version: Apache/2.4.6 (CentOS)
Server built:   Oct 19 2017 20:39:16

2) I checked the status using the following command,

# systemctl status httpd -l
● httpd.service - The Apache HTTP Server
      Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled;
vendor
preset: disabled)
      Active: active (running) since Wed 2018-03-07 23:46:32 IST; 1min
55s
ago
        Docs: man:httpd(8)
              man:apachectl(8)
     Process: 4351 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited,
status=0/SUCCESS)
    Main PID: 4359 (httpd)
      Status: "Total requests: 264; Current requests/sec: 0.1; Current
traffic: 204 B/sec"
      CGroup: /system.slice/httpd.service
              ├─4359 /usr/sbin/httpd -DFOREGROUND
              ├─4360 /usr/sbin/httpd -DFOREGROUND
              ├─4362 /usr/sbin/httpd -DFOREGROUND
              ├─5100 /usr/sbin/httpd -DFOREGROUND
              ├─5386 /usr/sbin/httpd -DFOREGROUND
              ├─5415 /usr/sbin/httpd -DFOREGROUND
              └─5416 /usr/sbin/httpd -DFOREGROUND

3) Since the httpd is pointing to the path :
/usr/lib/systemd/system/httpd.service

vi /usr/lib/systemd/system/httpd.service

[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target
Documentation=man:httpd(8)
Documentation=man:apachectl(8)

[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/httpd
ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND
ExecReload=/usr/sbin/httpd $OPTIONS -k graceful
ExecStop=/bin/kill -WINCH ${MAINPID}
# We want systemd to give httpd some time to finish gracefully, but
still
want
# it to kill httpd after TimeoutStopSec if something went wrong during
the
# graceful stop. Normally, Systemd sends SIGTERM signal right after the
# ExecStop, which would kill httpd. We are sending useless SIGCONT here
to
give
# httpd time to finish.
KillSignal=SIGCONT
PrivateTmp=true

[Install]
WantedBy=multi-user.target


4) As per the above command I found the env file is available
'/etc/sysconfig/httpd'

vi /etc/sysconfig/httpd

#
# This file can be used to set additional environment variables for
# the httpd process, or pass additional options to the httpd
# executable.
#
# Note: With previous versions of httpd, the MPM could be changed by
# editing an "HTTPD" variable here.  With the current version, that
# variable is now ignored.  The MPM is a loadable module, and the
# choice of MPM can be changed by editing the configuration file
/etc/httpd/conf.modules.d/00-mpm.conf
#

#
# To pass additional options (for instance, -D definitions) to the
# httpd binary at startup, set OPTIONS here.
#
#OPTIONS=

#
# This setting ensures the httpd process is started in the "C" locale
# by default.  (Some modules will not behave correctly if
# case-sensitive string comparisons are performed in a different
# locale.)
#
LANG=C


5) As per the above command, I found that the conf fileis available in
the
path : /etc/httpd/conf.modules.d/00-mpm.conf

vi /etc/httpd/conf.modules.d/00-mpm.conf

# Select the MPM module which should be used by uncommenting exactly
# one of the following LoadModule lines:

# prefork MPM: Implements a non-threaded, pre-forking web server
# See: http://httpd.apache.org/docs/2.4/mod/prefork.html
#LoadModule mpm_prefork_module modules/mod_mpm_prefork.so

# worker MPM: Multi-Processing Module implementing a hybrid
# multi-threaded multi-process web server
# See: http://httpd.apache.org/docs/2.4/mod/worker.html
#
LoadModule mpm_worker_module modules/mod_mpm_worker.so

# event MPM: A variant of the worker MPM with the goal of consuming
# threads only for connections with active processing
# See: http://httpd.apache.org/docs/2.4/mod/event.html
#
#LoadModule mpm_event_module modules/mod_mpm_event.so

<IfModule mpm_worker_module>
      ServerLimit 1000
      MaxRequestWorkers 1000
</IfModule>



As per your comment, I enabled the 'LoadModule mpm_worker_module
modules/mod_mpm_worker.so' with the ServerLimit and MaxRequestWorkers as
1000 still I am facing the issue for the following command in apache
benchmark test.

Completed 100 requests
SSL handshake failed (5).
SSL handshake failed (5).
SSL handshake failed (5).
SSL handshake failed (5).
SSL handshake failed (5).
SSL handshake failed (5).


NOTE : It always scales when I have concurrent request below 400

What is wrong in this apache configuration, why SSL handshake is failing
for concurrent request above 400 ?

Thanks,
Hari






On Wed, Mar 7, 2018 at 9:20 PM, Juan Hernández <jhernand@redhat.com>
wrote:

It means that with the default configuration the Apache web server can't

serve more than 256 concurrent connections. This applies to any
application
that uses Apache as the web frontend, not just to oVirt. If you want to
change that you have to change the MaxRequestWorkers and ServerLimit
parameters, as explained here:

     https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m
axrequestworkers

So, go to your oVirt engine machine and create a
/etc/httpd/conf.d/my.conf
file with this content:

     MaxRequestWorkers 1000
     ServerLimit 1000

Then restart the Apache server:

     # systemctl restart httpd

Then your web server should be able to handle 1000 concurrent requests,
and you will probably start to find other limits, like the amount of
memory
and CPU that those 1000 Apache child processes will consume, the number
of
threads in the JBoss application server, the number of connections to
the
database, etc.

Let me insist a bit that if you base your benchmark solely on the
number
of concurrent requests or connections that the server can handle you
may
end up with meaningless results, as a real world application can/should
use
the server much better than that.

On 03/07/2018 04:33 PM, Hari Prasanth Loganathan wrote:

With the default configuration of the web server it is impossible to

handle
more than 256 *connections* simultaneously. I guess that "ab" is
opening a
connection for each concurrent request, so when you reach request 257
the
web server will just reject the connection, there is nothing that the
JBoss
can do about it; you have to increase the number of connections
supported
by the web server.

*So Does it mean that oVirt cannot serve more than 257 request? *

My question is, If its possible How to scale this and what is the
configuration we need to change?

Also, we are taking a benchmark in using oVirt, So I need to find the
maximum possible oVirt request. So please let me know the
configuration
tuning for oVirt to achieve maximum concurrent request.

Thanks,
Hari

On Wed, Mar 7, 2018 at 7:25 PM, Juan Hernández <jhernand@redhat.com>
wrote:

With the default configuration of the web server it is impossible to

handle more than 256 *connections* simultaneously. I guess that "ab"

is
opening a connection for each concurrent request, so when you reach
request
257 the web server will just reject the connection, there is nothing
that
the JBoss can do about it; you have to increase the number of
connections
supported by the web server.

Or else you may want to re-consider why you want to use 1000
simultaneous
connections. It may be OK for a performance test, but there are
better
ways
to squeeze performance. For example, you could consider using HTTP
pipelining, which is much more friendly for the server than so many
connections. This is what we use when we need to send a large number
of
requests from other systems. There are examples of how to do that
with
the
Python and Ruby SDKs here:

      Python:

https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/
examples/asynchronous_inventory.py

      Ruby:

https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/
sdk/examples/asynchronous_inventory.rb


On 03/07/2018 02:43 PM, Hari Prasanth Loganathan wrote:

Hi Juan,

Thanks for the response.

I agree web server can handle only limited number of concurrent
requests.
But Why it is failing with SSL handshake failure for few requests,
Can't
the JBOSS wait and serve the request? We can spare the delay but not
with
the request fails. So Is there a configuration in oVirt which can be
tuned
to achieve this?

Thanks,
Hari

On Wed, Mar 7, 2018 at 7:05 PM, Juan Hernández <jhernand@redhat.com

wrote:

The first thing you will need to change for such a test is the
number
of

simultaneous connections accepted by the Apache web server: by
default

the
max is 256. See the Apache documentation here:

       https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m
axrequestworkers

In addition I also suggest that you consider using the "worker"
multi-processing module instead of the "prefork", as it usually
works
better when talking to a Java application server, because it
re-uses
connections better.

On 03/07/2018 02:20 PM, Hari Prasanth Loganathan wrote:

Hi Team,


*Description of problem:*

I am trying to achieve 1000 concurrent request to oVirt. What are
the
tunable parameters to achieve this?

I tried to perform the benchmarking for ovirt engine using Apache
benchmark
using the same SSO token.

ab -n 1000 -c 500 -k -H "accept: application/json" -H
"Authorization:
Bearer SSOTOKEN" https://172.30.56.70/ovirt-engine/
<https://172.30.56.70/ovirt-engine/api/vms/5440271b-afb3-48b
b-9ff1-076fc07ebf50/statistics>

When the number of concurrent request is 500, we are getting more
than
100
failures with the following error,

SSL read failed (1) - closing connection
139620982339352:error:

NOTE: It is scaling for concurrent request below 500.

I used the profiler to get the memory and CPU and it seems very
less,

        PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM
   TIME+
COMMAND
30413 ovirt     20   0 4226664 882396   6776 S 126.0 23.0
27:48.53
java

Configuration of the machine in which Ovirt is deployed :

RAM - 4GB,
Hard disk - 100GB,
core processor - 2,
OS - Cent7.x.

In which 2GB is allocated to oVirt.


Version-Release number of selected component (if applicable):

4.2.2


How reproducible:

If the number of concurrent requests are above 500, we are easily
facing
this issue.


*Actual results:*

SSL read failed (1) - closing connection
139620982339352:error:

*Expected results:*

Request success.


Thanks,
Hari



_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users