[ovirt-users] Tunable parameters in ovirt engine
Juan Hernández
jhernand at redhat.com
Thu Mar 8 11:45:13 UTC 2018
I think the configuration is good. There may be some connections that
the server is closing before the client expected, but that is normal, in
my opinion.
On 03/08/2018 12:34 PM, Hari Prasanth Loganathan wrote:
> This is the only error message we received from ab.
>
> I googled it and found that it is due to the connection drop. So It would
> be Great, If you could check my Apache server configuration I shared in the
> thread and let me know your thoughts on this.
>
> Thanks,
> Hari
>
> On Thu, Mar 8, 2018 at 4:56 PM, Juan Hernández <jhernand at redhat.com> wrote:
>
>> But other than those SSL error messages, are the connections really
>> failing? Can you share the results reported by "ab"?
>>
>>
>> On 03/08/2018 12:16 PM, Hari Prasanth Loganathan wrote:
>>
>>> No Juan, It is not working with any benchmark / application tool. It fails
>>> with the same error SSL handshake failed (5).
>>>
>>> Could you let me know the configuration of Apache web server is correct?
>>>
>>> Thanks,
>>> Hari
>>>
>>> On Thu, Mar 8, 2018 at 1:08 AM, Juan Hernández <jhernand at redhat.com>
>>> wrote:
>>>
>>> If you are still having problems I am inclined to think that it is a
>>>> client issue. For example, I'd try to remove the "-k" option from the
>>>> "ab"
>>>> command. If you use keep alive the server may decide anyhow to close the
>>>> connection after certain number of requests, even if the client asks to
>>>> keep it alive. Some clients don't handle that perfectly, "ab" may have
>>>> that
>>>> problem. If that makes the SSL error messages disappear then I think you
>>>> can safely ignore them, and restore the "-k" option, if you want.
>>>>
>>>> On 03/07/2018 07:30 PM, Hari Prasanth Loganathan wrote:
>>>>
>>>> Thanks Juan for your response. Appreciate it.
>>>>> But for some reason still, I am facing the same SSL handshake failed
>>>>> (5).
>>>>> Could you please check this configuration and let me know the issue in
>>>>> my
>>>>> ovirt engine environment.
>>>>>
>>>>> *Configuration of Apache server:*
>>>>>
>>>>>
>>>>> 1) httpd version,
>>>>>
>>>>> # httpd -v
>>>>> Server version: Apache/2.4.6 (CentOS)
>>>>> Server built: Oct 19 2017 20:39:16
>>>>>
>>>>> 2) I checked the status using the following command,
>>>>>
>>>>> # systemctl status httpd -l
>>>>> ● httpd.service - The Apache HTTP Server
>>>>> Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled;
>>>>> vendor
>>>>> preset: disabled)
>>>>> Active: active (running) since Wed 2018-03-07 23:46:32 IST; 1min
>>>>> 55s
>>>>> ago
>>>>> Docs: man:httpd(8)
>>>>> man:apachectl(8)
>>>>> Process: 4351 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited,
>>>>> status=0/SUCCESS)
>>>>> Main PID: 4359 (httpd)
>>>>> Status: "Total requests: 264; Current requests/sec: 0.1; Current
>>>>> traffic: 204 B/sec"
>>>>> CGroup: /system.slice/httpd.service
>>>>> ├─4359 /usr/sbin/httpd -DFOREGROUND
>>>>> ├─4360 /usr/sbin/httpd -DFOREGROUND
>>>>> ├─4362 /usr/sbin/httpd -DFOREGROUND
>>>>> ├─5100 /usr/sbin/httpd -DFOREGROUND
>>>>> ├─5386 /usr/sbin/httpd -DFOREGROUND
>>>>> ├─5415 /usr/sbin/httpd -DFOREGROUND
>>>>> └─5416 /usr/sbin/httpd -DFOREGROUND
>>>>>
>>>>> 3) Since the httpd is pointing to the path :
>>>>> /usr/lib/systemd/system/httpd.service
>>>>>
>>>>> vi /usr/lib/systemd/system/httpd.service
>>>>>
>>>>> [Unit]
>>>>> Description=The Apache HTTP Server
>>>>> After=network.target remote-fs.target nss-lookup.target
>>>>> Documentation=man:httpd(8)
>>>>> Documentation=man:apachectl(8)
>>>>>
>>>>> [Service]
>>>>> Type=notify
>>>>> EnvironmentFile=/etc/sysconfig/httpd
>>>>> ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND
>>>>> ExecReload=/usr/sbin/httpd $OPTIONS -k graceful
>>>>> ExecStop=/bin/kill -WINCH ${MAINPID}
>>>>> # We want systemd to give httpd some time to finish gracefully, but
>>>>> still
>>>>> want
>>>>> # it to kill httpd after TimeoutStopSec if something went wrong during
>>>>> the
>>>>> # graceful stop. Normally, Systemd sends SIGTERM signal right after the
>>>>> # ExecStop, which would kill httpd. We are sending useless SIGCONT here
>>>>> to
>>>>> give
>>>>> # httpd time to finish.
>>>>> KillSignal=SIGCONT
>>>>> PrivateTmp=true
>>>>>
>>>>> [Install]
>>>>> WantedBy=multi-user.target
>>>>>
>>>>>
>>>>> 4) As per the above command I found the env file is available
>>>>> '/etc/sysconfig/httpd'
>>>>>
>>>>> vi /etc/sysconfig/httpd
>>>>>
>>>>> #
>>>>> # This file can be used to set additional environment variables for
>>>>> # the httpd process, or pass additional options to the httpd
>>>>> # executable.
>>>>> #
>>>>> # Note: With previous versions of httpd, the MPM could be changed by
>>>>> # editing an "HTTPD" variable here. With the current version, that
>>>>> # variable is now ignored. The MPM is a loadable module, and the
>>>>> # choice of MPM can be changed by editing the configuration file
>>>>> /etc/httpd/conf.modules.d/00-mpm.conf
>>>>> #
>>>>>
>>>>> #
>>>>> # To pass additional options (for instance, -D definitions) to the
>>>>> # httpd binary at startup, set OPTIONS here.
>>>>> #
>>>>> #OPTIONS=
>>>>>
>>>>> #
>>>>> # This setting ensures the httpd process is started in the "C" locale
>>>>> # by default. (Some modules will not behave correctly if
>>>>> # case-sensitive string comparisons are performed in a different
>>>>> # locale.)
>>>>> #
>>>>> LANG=C
>>>>>
>>>>>
>>>>> 5) As per the above command, I found that the conf fileis available in
>>>>> the
>>>>> path : /etc/httpd/conf.modules.d/00-mpm.conf
>>>>>
>>>>> vi /etc/httpd/conf.modules.d/00-mpm.conf
>>>>>
>>>>> # Select the MPM module which should be used by uncommenting exactly
>>>>> # one of the following LoadModule lines:
>>>>>
>>>>> # prefork MPM: Implements a non-threaded, pre-forking web server
>>>>> # See: http://httpd.apache.org/docs/2.4/mod/prefork.html
>>>>> #LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
>>>>>
>>>>> # worker MPM: Multi-Processing Module implementing a hybrid
>>>>> # multi-threaded multi-process web server
>>>>> # See: http://httpd.apache.org/docs/2.4/mod/worker.html
>>>>> #
>>>>> LoadModule mpm_worker_module modules/mod_mpm_worker.so
>>>>>
>>>>> # event MPM: A variant of the worker MPM with the goal of consuming
>>>>> # threads only for connections with active processing
>>>>> # See: http://httpd.apache.org/docs/2.4/mod/event.html
>>>>> #
>>>>> #LoadModule mpm_event_module modules/mod_mpm_event.so
>>>>>
>>>>> <IfModule mpm_worker_module>
>>>>> ServerLimit 1000
>>>>> MaxRequestWorkers 1000
>>>>> </IfModule>
>>>>>
>>>>>
>>>>>
>>>>> As per your comment, I enabled the 'LoadModule mpm_worker_module
>>>>> modules/mod_mpm_worker.so' with the ServerLimit and MaxRequestWorkers as
>>>>> 1000 still I am facing the issue for the following command in apache
>>>>> benchmark test.
>>>>>
>>>>> Completed 100 requests
>>>>> SSL handshake failed (5).
>>>>> SSL handshake failed (5).
>>>>> SSL handshake failed (5).
>>>>> SSL handshake failed (5).
>>>>> SSL handshake failed (5).
>>>>> SSL handshake failed (5).
>>>>>
>>>>>
>>>>> NOTE : It always scales when I have concurrent request below 400
>>>>>
>>>>> What is wrong in this apache configuration, why SSL handshake is failing
>>>>> for concurrent request above 400 ?
>>>>>
>>>>> Thanks,
>>>>> Hari
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Mar 7, 2018 at 9:20 PM, Juan Hernández <jhernand at redhat.com>
>>>>> wrote:
>>>>>
>>>>> It means that with the default configuration the Apache web server can't
>>>>>
>>>>>> serve more than 256 concurrent connections. This applies to any
>>>>>> application
>>>>>> that uses Apache as the web frontend, not just to oVirt. If you want to
>>>>>> change that you have to change the MaxRequestWorkers and ServerLimit
>>>>>> parameters, as explained here:
>>>>>>
>>>>>> https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m
>>>>>> axrequestworkers
>>>>>>
>>>>>> So, go to your oVirt engine machine and create a
>>>>>> /etc/httpd/conf.d/my.conf
>>>>>> file with this content:
>>>>>>
>>>>>> MaxRequestWorkers 1000
>>>>>> ServerLimit 1000
>>>>>>
>>>>>> Then restart the Apache server:
>>>>>>
>>>>>> # systemctl restart httpd
>>>>>>
>>>>>> Then your web server should be able to handle 1000 concurrent requests,
>>>>>> and you will probably start to find other limits, like the amount of
>>>>>> memory
>>>>>> and CPU that those 1000 Apache child processes will consume, the number
>>>>>> of
>>>>>> threads in the JBoss application server, the number of connections to
>>>>>> the
>>>>>> database, etc.
>>>>>>
>>>>>> Let me insist a bit that if you base your benchmark solely on the
>>>>>> number
>>>>>> of concurrent requests or connections that the server can handle you
>>>>>> may
>>>>>> end up with meaningless results, as a real world application can/should
>>>>>> use
>>>>>> the server much better than that.
>>>>>>
>>>>>> On 03/07/2018 04:33 PM, Hari Prasanth Loganathan wrote:
>>>>>>
>>>>>> With the default configuration of the web server it is impossible to
>>>>>>
>>>>>>> handle
>>>>>>> more than 256 *connections* simultaneously. I guess that "ab" is
>>>>>>> opening a
>>>>>>> connection for each concurrent request, so when you reach request 257
>>>>>>> the
>>>>>>> web server will just reject the connection, there is nothing that the
>>>>>>> JBoss
>>>>>>> can do about it; you have to increase the number of connections
>>>>>>> supported
>>>>>>> by the web server.
>>>>>>>
>>>>>>> *So Does it mean that oVirt cannot serve more than 257 request? *
>>>>>>>
>>>>>>> My question is, If its possible How to scale this and what is the
>>>>>>> configuration we need to change?
>>>>>>>
>>>>>>> Also, we are taking a benchmark in using oVirt, So I need to find the
>>>>>>> maximum possible oVirt request. So please let me know the
>>>>>>> configuration
>>>>>>> tuning for oVirt to achieve maximum concurrent request.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Hari
>>>>>>>
>>>>>>> On Wed, Mar 7, 2018 at 7:25 PM, Juan Hernández <jhernand at redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> With the default configuration of the web server it is impossible to
>>>>>>>
>>>>>>> handle more than 256 *connections* simultaneously. I guess that "ab"
>>>>>>>> is
>>>>>>>> opening a connection for each concurrent request, so when you reach
>>>>>>>> request
>>>>>>>> 257 the web server will just reject the connection, there is nothing
>>>>>>>> that
>>>>>>>> the JBoss can do about it; you have to increase the number of
>>>>>>>> connections
>>>>>>>> supported by the web server.
>>>>>>>>
>>>>>>>> Or else you may want to re-consider why you want to use 1000
>>>>>>>> simultaneous
>>>>>>>> connections. It may be OK for a performance test, but there are
>>>>>>>> better
>>>>>>>> ways
>>>>>>>> to squeeze performance. For example, you could consider using HTTP
>>>>>>>> pipelining, which is much more friendly for the server than so many
>>>>>>>> connections. This is what we use when we need to send a large number
>>>>>>>> of
>>>>>>>> requests from other systems. There are examples of how to do that
>>>>>>>> with
>>>>>>>> the
>>>>>>>> Python and Ruby SDKs here:
>>>>>>>>
>>>>>>>> Python:
>>>>>>>>
>>>>>>>> https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/
>>>>>>>> examples/asynchronous_inventory.py
>>>>>>>>
>>>>>>>> Ruby:
>>>>>>>>
>>>>>>>> https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/
>>>>>>>> sdk/examples/asynchronous_inventory.rb
>>>>>>>>
>>>>>>>>
>>>>>>>> On 03/07/2018 02:43 PM, Hari Prasanth Loganathan wrote:
>>>>>>>>
>>>>>>>> Hi Juan,
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks for the response.
>>>>>>>>>
>>>>>>>>> I agree web server can handle only limited number of concurrent
>>>>>>>>> requests.
>>>>>>>>> But Why it is failing with SSL handshake failure for few requests,
>>>>>>>>> Can't
>>>>>>>>> the JBOSS wait and serve the request? We can spare the delay but not
>>>>>>>>> with
>>>>>>>>> the request fails. So Is there a configuration in oVirt which can be
>>>>>>>>> tuned
>>>>>>>>> to achieve this?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Hari
>>>>>>>>>
>>>>>>>>> On Wed, Mar 7, 2018 at 7:05 PM, Juan Hernández <jhernand at redhat.com
>>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> The first thing you will need to change for such a test is the
>>>>>>>>> number
>>>>>>>>> of
>>>>>>>>>
>>>>>>>>> simultaneous connections accepted by the Apache web server: by
>>>>>>>>> default
>>>>>>>>>
>>>>>>>>>> the
>>>>>>>>>> max is 256. See the Apache documentation here:
>>>>>>>>>>
>>>>>>>>>> https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m
>>>>>>>>>> axrequestworkers
>>>>>>>>>>
>>>>>>>>>> In addition I also suggest that you consider using the "worker"
>>>>>>>>>> multi-processing module instead of the "prefork", as it usually
>>>>>>>>>> works
>>>>>>>>>> better when talking to a Java application server, because it
>>>>>>>>>> re-uses
>>>>>>>>>> connections better.
>>>>>>>>>>
>>>>>>>>>> On 03/07/2018 02:20 PM, Hari Prasanth Loganathan wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Team,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *Description of problem:*
>>>>>>>>>>>
>>>>>>>>>>> I am trying to achieve 1000 concurrent request to oVirt. What are
>>>>>>>>>>> the
>>>>>>>>>>> tunable parameters to achieve this?
>>>>>>>>>>>
>>>>>>>>>>> I tried to perform the benchmarking for ovirt engine using Apache
>>>>>>>>>>> benchmark
>>>>>>>>>>> using the same SSO token.
>>>>>>>>>>>
>>>>>>>>>>> ab -n 1000 -c 500 -k -H "accept: application/json" -H
>>>>>>>>>>> "Authorization:
>>>>>>>>>>> Bearer SSOTOKEN" https://172.30.56.70/ovirt-engine/
>>>>>>>>>>> <https://172.30.56.70/ovirt-engine/api/vms/5440271b-afb3-48b
>>>>>>>>>>> b-9ff1-076fc07ebf50/statistics>
>>>>>>>>>>>
>>>>>>>>>>> When the number of concurrent request is 500, we are getting more
>>>>>>>>>>> than
>>>>>>>>>>> 100
>>>>>>>>>>> failures with the following error,
>>>>>>>>>>>
>>>>>>>>>>> SSL read failed (1) - closing connection
>>>>>>>>>>> 139620982339352:error:
>>>>>>>>>>>
>>>>>>>>>>> NOTE: It is scaling for concurrent request below 500.
>>>>>>>>>>>
>>>>>>>>>>> I used the profiler to get the memory and CPU and it seems very
>>>>>>>>>>> less,
>>>>>>>>>>>
>>>>>>>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM
>>>>>>>>>>> TIME+
>>>>>>>>>>> COMMAND
>>>>>>>>>>> 30413 ovirt 20 0 4226664 882396 6776 S 126.0 23.0
>>>>>>>>>>> 27:48.53
>>>>>>>>>>> java
>>>>>>>>>>>
>>>>>>>>>>> Configuration of the machine in which Ovirt is deployed :
>>>>>>>>>>>
>>>>>>>>>>> RAM - 4GB,
>>>>>>>>>>> Hard disk - 100GB,
>>>>>>>>>>> core processor - 2,
>>>>>>>>>>> OS - Cent7.x.
>>>>>>>>>>>
>>>>>>>>>>> In which 2GB is allocated to oVirt.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Version-Release number of selected component (if applicable):
>>>>>>>>>>>
>>>>>>>>>>> 4.2.2
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> How reproducible:
>>>>>>>>>>>
>>>>>>>>>>> If the number of concurrent requests are above 500, we are easily
>>>>>>>>>>> facing
>>>>>>>>>>> this issue.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *Actual results:*
>>>>>>>>>>>
>>>>>>>>>>> SSL read failed (1) - closing connection
>>>>>>>>>>> 139620982339352:error:
>>>>>>>>>>>
>>>>>>>>>>> *Expected results:*
>>>>>>>>>>>
>>>>>>>>>>> Request success.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Hari
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Users mailing list
>>>>>>>>>>> Users at ovirt.org
>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
More information about the Users
mailing list