[ovirt-users] Tunable parameters in ovirt engine
Juan Hernández
jhernand at redhat.com
Thu Mar 8 11:26:25 UTC 2018
But other than those SSL error messages, are the connections really
failing? Can you share the results reported by "ab"?
On 03/08/2018 12:16 PM, Hari Prasanth Loganathan wrote:
> No Juan, It is not working with any benchmark / application tool. It fails
> with the same error SSL handshake failed (5).
>
> Could you let me know the configuration of Apache web server is correct?
>
> Thanks,
> Hari
>
> On Thu, Mar 8, 2018 at 1:08 AM, Juan Hernández <jhernand at redhat.com> wrote:
>
>> If you are still having problems I am inclined to think that it is a
>> client issue. For example, I'd try to remove the "-k" option from the "ab"
>> command. If you use keep alive the server may decide anyhow to close the
>> connection after certain number of requests, even if the client asks to
>> keep it alive. Some clients don't handle that perfectly, "ab" may have that
>> problem. If that makes the SSL error messages disappear then I think you
>> can safely ignore them, and restore the "-k" option, if you want.
>>
>> On 03/07/2018 07:30 PM, Hari Prasanth Loganathan wrote:
>>
>>> Thanks Juan for your response. Appreciate it.
>>> But for some reason still, I am facing the same SSL handshake failed (5).
>>> Could you please check this configuration and let me know the issue in my
>>> ovirt engine environment.
>>>
>>> *Configuration of Apache server:*
>>>
>>>
>>> 1) httpd version,
>>>
>>> # httpd -v
>>> Server version: Apache/2.4.6 (CentOS)
>>> Server built: Oct 19 2017 20:39:16
>>>
>>> 2) I checked the status using the following command,
>>>
>>> # systemctl status httpd -l
>>> ● httpd.service - The Apache HTTP Server
>>> Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled;
>>> vendor
>>> preset: disabled)
>>> Active: active (running) since Wed 2018-03-07 23:46:32 IST; 1min 55s
>>> ago
>>> Docs: man:httpd(8)
>>> man:apachectl(8)
>>> Process: 4351 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited,
>>> status=0/SUCCESS)
>>> Main PID: 4359 (httpd)
>>> Status: "Total requests: 264; Current requests/sec: 0.1; Current
>>> traffic: 204 B/sec"
>>> CGroup: /system.slice/httpd.service
>>> ├─4359 /usr/sbin/httpd -DFOREGROUND
>>> ├─4360 /usr/sbin/httpd -DFOREGROUND
>>> ├─4362 /usr/sbin/httpd -DFOREGROUND
>>> ├─5100 /usr/sbin/httpd -DFOREGROUND
>>> ├─5386 /usr/sbin/httpd -DFOREGROUND
>>> ├─5415 /usr/sbin/httpd -DFOREGROUND
>>> └─5416 /usr/sbin/httpd -DFOREGROUND
>>>
>>> 3) Since the httpd is pointing to the path :
>>> /usr/lib/systemd/system/httpd.service
>>>
>>> vi /usr/lib/systemd/system/httpd.service
>>>
>>> [Unit]
>>> Description=The Apache HTTP Server
>>> After=network.target remote-fs.target nss-lookup.target
>>> Documentation=man:httpd(8)
>>> Documentation=man:apachectl(8)
>>>
>>> [Service]
>>> Type=notify
>>> EnvironmentFile=/etc/sysconfig/httpd
>>> ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND
>>> ExecReload=/usr/sbin/httpd $OPTIONS -k graceful
>>> ExecStop=/bin/kill -WINCH ${MAINPID}
>>> # We want systemd to give httpd some time to finish gracefully, but still
>>> want
>>> # it to kill httpd after TimeoutStopSec if something went wrong during the
>>> # graceful stop. Normally, Systemd sends SIGTERM signal right after the
>>> # ExecStop, which would kill httpd. We are sending useless SIGCONT here to
>>> give
>>> # httpd time to finish.
>>> KillSignal=SIGCONT
>>> PrivateTmp=true
>>>
>>> [Install]
>>> WantedBy=multi-user.target
>>>
>>>
>>> 4) As per the above command I found the env file is available
>>> '/etc/sysconfig/httpd'
>>>
>>> vi /etc/sysconfig/httpd
>>>
>>> #
>>> # This file can be used to set additional environment variables for
>>> # the httpd process, or pass additional options to the httpd
>>> # executable.
>>> #
>>> # Note: With previous versions of httpd, the MPM could be changed by
>>> # editing an "HTTPD" variable here. With the current version, that
>>> # variable is now ignored. The MPM is a loadable module, and the
>>> # choice of MPM can be changed by editing the configuration file
>>> /etc/httpd/conf.modules.d/00-mpm.conf
>>> #
>>>
>>> #
>>> # To pass additional options (for instance, -D definitions) to the
>>> # httpd binary at startup, set OPTIONS here.
>>> #
>>> #OPTIONS=
>>>
>>> #
>>> # This setting ensures the httpd process is started in the "C" locale
>>> # by default. (Some modules will not behave correctly if
>>> # case-sensitive string comparisons are performed in a different
>>> # locale.)
>>> #
>>> LANG=C
>>>
>>>
>>> 5) As per the above command, I found that the conf fileis available in the
>>> path : /etc/httpd/conf.modules.d/00-mpm.conf
>>>
>>> vi /etc/httpd/conf.modules.d/00-mpm.conf
>>>
>>> # Select the MPM module which should be used by uncommenting exactly
>>> # one of the following LoadModule lines:
>>>
>>> # prefork MPM: Implements a non-threaded, pre-forking web server
>>> # See: http://httpd.apache.org/docs/2.4/mod/prefork.html
>>> #LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
>>>
>>> # worker MPM: Multi-Processing Module implementing a hybrid
>>> # multi-threaded multi-process web server
>>> # See: http://httpd.apache.org/docs/2.4/mod/worker.html
>>> #
>>> LoadModule mpm_worker_module modules/mod_mpm_worker.so
>>>
>>> # event MPM: A variant of the worker MPM with the goal of consuming
>>> # threads only for connections with active processing
>>> # See: http://httpd.apache.org/docs/2.4/mod/event.html
>>> #
>>> #LoadModule mpm_event_module modules/mod_mpm_event.so
>>>
>>> <IfModule mpm_worker_module>
>>> ServerLimit 1000
>>> MaxRequestWorkers 1000
>>> </IfModule>
>>>
>>>
>>>
>>> As per your comment, I enabled the 'LoadModule mpm_worker_module
>>> modules/mod_mpm_worker.so' with the ServerLimit and MaxRequestWorkers as
>>> 1000 still I am facing the issue for the following command in apache
>>> benchmark test.
>>>
>>> Completed 100 requests
>>> SSL handshake failed (5).
>>> SSL handshake failed (5).
>>> SSL handshake failed (5).
>>> SSL handshake failed (5).
>>> SSL handshake failed (5).
>>> SSL handshake failed (5).
>>>
>>>
>>> NOTE : It always scales when I have concurrent request below 400
>>>
>>> What is wrong in this apache configuration, why SSL handshake is failing
>>> for concurrent request above 400 ?
>>>
>>> Thanks,
>>> Hari
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Mar 7, 2018 at 9:20 PM, Juan Hernández <jhernand at redhat.com>
>>> wrote:
>>>
>>> It means that with the default configuration the Apache web server can't
>>>> serve more than 256 concurrent connections. This applies to any
>>>> application
>>>> that uses Apache as the web frontend, not just to oVirt. If you want to
>>>> change that you have to change the MaxRequestWorkers and ServerLimit
>>>> parameters, as explained here:
>>>>
>>>> https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m
>>>> axrequestworkers
>>>>
>>>> So, go to your oVirt engine machine and create a
>>>> /etc/httpd/conf.d/my.conf
>>>> file with this content:
>>>>
>>>> MaxRequestWorkers 1000
>>>> ServerLimit 1000
>>>>
>>>> Then restart the Apache server:
>>>>
>>>> # systemctl restart httpd
>>>>
>>>> Then your web server should be able to handle 1000 concurrent requests,
>>>> and you will probably start to find other limits, like the amount of
>>>> memory
>>>> and CPU that those 1000 Apache child processes will consume, the number
>>>> of
>>>> threads in the JBoss application server, the number of connections to the
>>>> database, etc.
>>>>
>>>> Let me insist a bit that if you base your benchmark solely on the number
>>>> of concurrent requests or connections that the server can handle you may
>>>> end up with meaningless results, as a real world application can/should
>>>> use
>>>> the server much better than that.
>>>>
>>>> On 03/07/2018 04:33 PM, Hari Prasanth Loganathan wrote:
>>>>
>>>> With the default configuration of the web server it is impossible to
>>>>> handle
>>>>> more than 256 *connections* simultaneously. I guess that "ab" is
>>>>> opening a
>>>>> connection for each concurrent request, so when you reach request 257
>>>>> the
>>>>> web server will just reject the connection, there is nothing that the
>>>>> JBoss
>>>>> can do about it; you have to increase the number of connections
>>>>> supported
>>>>> by the web server.
>>>>>
>>>>> *So Does it mean that oVirt cannot serve more than 257 request? *
>>>>>
>>>>> My question is, If its possible How to scale this and what is the
>>>>> configuration we need to change?
>>>>>
>>>>> Also, we are taking a benchmark in using oVirt, So I need to find the
>>>>> maximum possible oVirt request. So please let me know the configuration
>>>>> tuning for oVirt to achieve maximum concurrent request.
>>>>>
>>>>> Thanks,
>>>>> Hari
>>>>>
>>>>> On Wed, Mar 7, 2018 at 7:25 PM, Juan Hernández <jhernand at redhat.com>
>>>>> wrote:
>>>>>
>>>>> With the default configuration of the web server it is impossible to
>>>>>
>>>>>> handle more than 256 *connections* simultaneously. I guess that "ab" is
>>>>>> opening a connection for each concurrent request, so when you reach
>>>>>> request
>>>>>> 257 the web server will just reject the connection, there is nothing
>>>>>> that
>>>>>> the JBoss can do about it; you have to increase the number of
>>>>>> connections
>>>>>> supported by the web server.
>>>>>>
>>>>>> Or else you may want to re-consider why you want to use 1000
>>>>>> simultaneous
>>>>>> connections. It may be OK for a performance test, but there are better
>>>>>> ways
>>>>>> to squeeze performance. For example, you could consider using HTTP
>>>>>> pipelining, which is much more friendly for the server than so many
>>>>>> connections. This is what we use when we need to send a large number of
>>>>>> requests from other systems. There are examples of how to do that with
>>>>>> the
>>>>>> Python and Ruby SDKs here:
>>>>>>
>>>>>> Python:
>>>>>>
>>>>>> https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/
>>>>>> examples/asynchronous_inventory.py
>>>>>>
>>>>>> Ruby:
>>>>>>
>>>>>> https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/
>>>>>> sdk/examples/asynchronous_inventory.rb
>>>>>>
>>>>>>
>>>>>> On 03/07/2018 02:43 PM, Hari Prasanth Loganathan wrote:
>>>>>>
>>>>>> Hi Juan,
>>>>>>
>>>>>>>
>>>>>>> Thanks for the response.
>>>>>>>
>>>>>>> I agree web server can handle only limited number of concurrent
>>>>>>> requests.
>>>>>>> But Why it is failing with SSL handshake failure for few requests,
>>>>>>> Can't
>>>>>>> the JBOSS wait and serve the request? We can spare the delay but not
>>>>>>> with
>>>>>>> the request fails. So Is there a configuration in oVirt which can be
>>>>>>> tuned
>>>>>>> to achieve this?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Hari
>>>>>>>
>>>>>>> On Wed, Mar 7, 2018 at 7:05 PM, Juan Hernández <jhernand at redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> The first thing you will need to change for such a test is the number
>>>>>>> of
>>>>>>>
>>>>>>> simultaneous connections accepted by the Apache web server: by default
>>>>>>>> the
>>>>>>>> max is 256. See the Apache documentation here:
>>>>>>>>
>>>>>>>> https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m
>>>>>>>> axrequestworkers
>>>>>>>>
>>>>>>>> In addition I also suggest that you consider using the "worker"
>>>>>>>> multi-processing module instead of the "prefork", as it usually works
>>>>>>>> better when talking to a Java application server, because it re-uses
>>>>>>>> connections better.
>>>>>>>>
>>>>>>>> On 03/07/2018 02:20 PM, Hari Prasanth Loganathan wrote:
>>>>>>>>
>>>>>>>> Hi Team,
>>>>>>>>
>>>>>>>>
>>>>>>>>> *Description of problem:*
>>>>>>>>>
>>>>>>>>> I am trying to achieve 1000 concurrent request to oVirt. What are
>>>>>>>>> the
>>>>>>>>> tunable parameters to achieve this?
>>>>>>>>>
>>>>>>>>> I tried to perform the benchmarking for ovirt engine using Apache
>>>>>>>>> benchmark
>>>>>>>>> using the same SSO token.
>>>>>>>>>
>>>>>>>>> ab -n 1000 -c 500 -k -H "accept: application/json" -H
>>>>>>>>> "Authorization:
>>>>>>>>> Bearer SSOTOKEN" https://172.30.56.70/ovirt-engine/
>>>>>>>>> <https://172.30.56.70/ovirt-engine/api/vms/5440271b-afb3-48b
>>>>>>>>> b-9ff1-076fc07ebf50/statistics>
>>>>>>>>>
>>>>>>>>> When the number of concurrent request is 500, we are getting more
>>>>>>>>> than
>>>>>>>>> 100
>>>>>>>>> failures with the following error,
>>>>>>>>>
>>>>>>>>> SSL read failed (1) - closing connection
>>>>>>>>> 139620982339352:error:
>>>>>>>>>
>>>>>>>>> NOTE: It is scaling for concurrent request below 500.
>>>>>>>>>
>>>>>>>>> I used the profiler to get the memory and CPU and it seems very
>>>>>>>>> less,
>>>>>>>>>
>>>>>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM
>>>>>>>>> TIME+
>>>>>>>>> COMMAND
>>>>>>>>> 30413 ovirt 20 0 4226664 882396 6776 S 126.0 23.0 27:48.53
>>>>>>>>> java
>>>>>>>>>
>>>>>>>>> Configuration of the machine in which Ovirt is deployed :
>>>>>>>>>
>>>>>>>>> RAM - 4GB,
>>>>>>>>> Hard disk - 100GB,
>>>>>>>>> core processor - 2,
>>>>>>>>> OS - Cent7.x.
>>>>>>>>>
>>>>>>>>> In which 2GB is allocated to oVirt.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Version-Release number of selected component (if applicable):
>>>>>>>>>
>>>>>>>>> 4.2.2
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> How reproducible:
>>>>>>>>>
>>>>>>>>> If the number of concurrent requests are above 500, we are easily
>>>>>>>>> facing
>>>>>>>>> this issue.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *Actual results:*
>>>>>>>>>
>>>>>>>>> SSL read failed (1) - closing connection
>>>>>>>>> 139620982339352:error:
>>>>>>>>>
>>>>>>>>> *Expected results:*
>>>>>>>>>
>>>>>>>>> Request success.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Hari
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Users mailing list
>>>>>>>>> Users at ovirt.org
>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
More information about the Users
mailing list