Re: [ovirt-users] Tunable parameters in ovirt engine

I think the configuration is good. There may be some connections that the server is closing before the client expected, but that is normal, in my opinion. On 03/08/2018 12:34 PM, Hari Prasanth Loganathan wrote: > This is the only error message we received from ab. > > I googled it and found that it is due to the connection drop. So It would > be Great, If you could check my Apache server configuration I shared in > the > thread and let me know your thoughts on this. > > Thanks, > Hari > > On Thu, Mar 8, 2018 at 4:56 PM, Juan Hernández <jhernand(a)redhat.com&gt; > wrote: > > But other than those SSL error messages, are the connections really >> failing? Can you share the results reported by "ab"? >> >> >> On 03/08/2018 12:16 PM, Hari Prasanth Loganathan wrote: >> >> No Juan, It is not working with any benchmark / application tool. It >>> fails >>> with the same error SSL handshake failed (5). >>> >>> Could you let me know the configuration of Apache web server is correct? >>> >>> Thanks, >>> Hari >>> >>> On Thu, Mar 8, 2018 at 1:08 AM, Juan Hernández <jhernand(a)redhat.com&gt; >>> wrote: >>> >>> If you are still having problems I am inclined to think that it is a >>> >>>> client issue. For example, I'd try to remove the "-k" option from the >>>> "ab" >>>> command. If you use keep alive the server may decide anyhow to close >>>> the >>>> connection after certain number of requests, even if the client asks to >>>> keep it alive. Some clients don't handle that perfectly, "ab" may have >>>> that >>>> problem. If that makes the SSL error messages disappear then I think >>>> you >>>> can safely ignore them, and restore the "-k" option, if you want. >>>> >>>> On 03/07/2018 07:30 PM, Hari Prasanth Loganathan wrote: >>>> >>>> Thanks Juan for your response. Appreciate it. >>>> >>>>> But for some reason still, I am facing the same SSL handshake failed >>>>> (5). >>>>> Could you please check this configuration and let me know the issue in >>>>> my >>>>> ovirt engine environment. >>>>> >>>>> *Configuration of Apache server:* >>>>> >>>>> >>>>> 1) httpd version, >>>>> >>>>> # httpd -v >>>>> Server version: Apache/2.4.6 (CentOS) >>>>> Server built: Oct 19 2017 20:39:16 >>>>> >>>>> 2) I checked the status using the following command, >>>>> >>>>> # systemctl status httpd -l >>>>> ● httpd.service - The Apache HTTP Server >>>>> Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; >>>>> vendor >>>>> preset: disabled) >>>>> Active: active (running) since Wed 2018-03-07 23:46:32 IST; 1min >>>>> 55s >>>>> ago >>>>> Docs: man:httpd(8) >>>>> man:apachectl(8) >>>>> Process: 4351 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, >>>>> status=0/SUCCESS) >>>>> Main PID: 4359 (httpd) >>>>> Status: "Total requests: 264; Current requests/sec: 0.1; Current >>>>> traffic: 204 B/sec" >>>>> CGroup: /system.slice/httpd.service >>>>> ├─4359 /usr/sbin/httpd -DFOREGROUND >>>>> ├─4360 /usr/sbin/httpd -DFOREGROUND >>>>> ├─4362 /usr/sbin/httpd -DFOREGROUND >>>>> ├─5100 /usr/sbin/httpd -DFOREGROUND >>>>> ├─5386 /usr/sbin/httpd -DFOREGROUND >>>>> ├─5415 /usr/sbin/httpd -DFOREGROUND >>>>> └─5416 /usr/sbin/httpd -DFOREGROUND >>>>> >>>>> 3) Since the httpd is pointing to the path : >>>>> /usr/lib/systemd/system/httpd.service >>>>> >>>>> vi /usr/lib/systemd/system/httpd.service >>>>> >>>>> [Unit] >>>>> Description=The Apache HTTP Server >>>>> After=network.target remote-fs.target nss-lookup.target >>>>> Documentation=man:httpd(8) >>>>> Documentation=man:apachectl(8) >>>>> >>>>> [Service] >>>>> Type=notify >>>>> EnvironmentFile=/etc/sysconfig/httpd >>>>> ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND >>>>> ExecReload=/usr/sbin/httpd $OPTIONS -k graceful >>>>> ExecStop=/bin/kill -WINCH ${MAINPID} >>>>> # We want systemd to give httpd some time to finish gracefully, but >>>>> still >>>>> want >>>>> # it to kill httpd after TimeoutStopSec if something went wrong during >>>>> the >>>>> # graceful stop. Normally, Systemd sends SIGTERM signal right after >>>>> the >>>>> # ExecStop, which would kill httpd. We are sending useless SIGCONT >>>>> here >>>>> to >>>>> give >>>>> # httpd time to finish. >>>>> KillSignal=SIGCONT >>>>> PrivateTmp=true >>>>> >>>>> [Install] >>>>> WantedBy=multi-user.target >>>>> >>>>> >>>>> 4) As per the above command I found the env file is available >>>>> '/etc/sysconfig/httpd' >>>>> >>>>> vi /etc/sysconfig/httpd >>>>> >>>>> # >>>>> # This file can be used to set additional environment variables for >>>>> # the httpd process, or pass additional options to the httpd >>>>> # executable. >>>>> # >>>>> # Note: With previous versions of httpd, the MPM could be changed by >>>>> # editing an "HTTPD" variable here. With the current version, that >>>>> # variable is now ignored. The MPM is a loadable module, and the >>>>> # choice of MPM can be changed by editing the configuration file >>>>> /etc/httpd/conf.modules.d/00-mpm.conf >>>>> # >>>>> >>>>> # >>>>> # To pass additional options (for instance, -D definitions) to the >>>>> # httpd binary at startup, set OPTIONS here. >>>>> # >>>>> #OPTIONS= >>>>> >>>>> # >>>>> # This setting ensures the httpd process is started in the "C" locale >>>>> # by default. (Some modules will not behave correctly if >>>>> # case-sensitive string comparisons are performed in a different >>>>> # locale.) >>>>> # >>>>> LANG=C >>>>> >>>>> >>>>> 5) As per the above command, I found that the conf fileis available in >>>>> the >>>>> path : /etc/httpd/conf.modules.d/00-mpm.conf >>>>> >>>>> vi /etc/httpd/conf.modules.d/00-mpm.conf >>>>> >>>>> # Select the MPM module which should be used by uncommenting exactly >>>>> # one of the following LoadModule lines: >>>>> >>>>> # prefork MPM: Implements a non-threaded, pre-forking web server >>>>> # See: http://httpd.apache.org/docs/2.4/mod/prefork.html >>>>> #LoadModule mpm_prefork_module modules/mod_mpm_prefork.so >>>>> >>>>> # worker MPM: Multi-Processing Module implementing a hybrid >>>>> # multi-threaded multi-process web server >>>>> # See: http://httpd.apache.org/docs/2.4/mod/worker.html >>>>> # >>>>> LoadModule mpm_worker_module modules/mod_mpm_worker.so >>>>> >>>>> # event MPM: A variant of the worker MPM with the goal of consuming >>>>> # threads only for connections with active processing >>>>> # See: http://httpd.apache.org/docs/2.4/mod/event.html >>>>> # >>>>> #LoadModule mpm_event_module modules/mod_mpm_event.so >>>>> >>>>> <IfModule mpm_worker_module> >>>>> ServerLimit 1000 >>>>> MaxRequestWorkers 1000 >>>>> </IfModule> >>>>> >>>>> >>>>> >>>>> As per your comment, I enabled the 'LoadModule mpm_worker_module >>>>> modules/mod_mpm_worker.so' with the ServerLimit and MaxRequestWorkers >>>>> as >>>>> 1000 still I am facing the issue for the following command in apache >>>>> benchmark test. >>>>> >>>>> Completed 100 requests >>>>> SSL handshake failed (5). >>>>> SSL handshake failed (5). >>>>> SSL handshake failed (5). >>>>> SSL handshake failed (5). >>>>> SSL handshake failed (5). >>>>> SSL handshake failed (5). >>>>> >>>>> >>>>> NOTE : It always scales when I have concurrent request below 400 >>>>> >>>>> What is wrong in this apache configuration, why SSL handshake is >>>>> failing >>>>> for concurrent request above 400 ? >>>>> >>>>> Thanks, >>>>> Hari >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Mar 7, 2018 at 9:20 PM, Juan Hernández <jhernand(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>> It means that with the default configuration the Apache web server >>>>> can't >>>>> >>>>> serve more than 256 concurrent connections. This applies to any >>>>>> application >>>>>> that uses Apache as the web frontend, not just to oVirt. If you want >>>>>> to >>>>>> change that you have to change the MaxRequestWorkers and ServerLimit >>>>>> parameters, as explained here: >>>>>> >>>>>> https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m >>>>>> axrequestworkers >>>>>> >>>>>> So, go to your oVirt engine machine and create a >>>>>> /etc/httpd/conf.d/my.conf >>>>>> file with this content: >>>>>> >>>>>> MaxRequestWorkers 1000 >>>>>> ServerLimit 1000 >>>>>> >>>>>> Then restart the Apache server: >>>>>> >>>>>> # systemctl restart httpd >>>>>> >>>>>> Then your web server should be able to handle 1000 concurrent >>>>>> requests, >>>>>> and you will probably start to find other limits, like the amount of >>>>>> memory >>>>>> and CPU that those 1000 Apache child processes will consume, the >>>>>> number >>>>>> of >>>>>> threads in the JBoss application server, the number of connections to >>>>>> the >>>>>> database, etc. >>>>>> >>>>>> Let me insist a bit that if you base your benchmark solely on the >>>>>> number >>>>>> of concurrent requests or connections that the server can handle you >>>>>> may >>>>>> end up with meaningless results, as a real world application >>>>>> can/should >>>>>> use >>>>>> the server much better than that. >>>>>> >>>>>> On 03/07/2018 04:33 PM, Hari Prasanth Loganathan wrote: >>>>>> >>>>>> With the default configuration of the web server it is impossible to >>>>>> >>>>>> handle >>>>>>> more than 256 *connections* simultaneously. I guess that "ab" is >>>>>>> opening a >>>>>>> connection for each concurrent request, so when you reach request >>>>>>> 257 >>>>>>> the >>>>>>> web server will just reject the connection, there is nothing that >>>>>>> the >>>>>>> JBoss >>>>>>> can do about it; you have to increase the number of connections >>>>>>> supported >>>>>>> by the web server. >>>>>>> >>>>>>> *So Does it mean that oVirt cannot serve more than 257 request? * >>>>>>> >>>>>>> My question is, If its possible How to scale this and what is the >>>>>>> configuration we need to change? >>>>>>> >>>>>>> Also, we are taking a benchmark in using oVirt, So I need to find >>>>>>> the >>>>>>> maximum possible oVirt request. So please let me know the >>>>>>> configuration >>>>>>> tuning for oVirt to achieve maximum concurrent request. >>>>>>> >>>>>>> Thanks, >>>>>>> Hari >>>>>>> >>>>>>> On Wed, Mar 7, 2018 at 7:25 PM, Juan Hernández <jhernand(a)redhat.com >>>>>>> > >>>>>>> wrote: >>>>>>> >>>>>>> With the default configuration of the web server it is impossible to >>>>>>> >>>>>>> handle more than 256 *connections* simultaneously. I guess that "ab" >>>>>>> >>>>>>>> is >>>>>>>> opening a connection for each concurrent request, so when you reach >>>>>>>> request >>>>>>>> 257 the web server will just reject the connection, there is >>>>>>>> nothing >>>>>>>> that >>>>>>>> the JBoss can do about it; you have to increase the number of >>>>>>>> connections >>>>>>>> supported by the web server. >>>>>>>> >>>>>>>> Or else you may want to re-consider why you want to use 1000 >>>>>>>> simultaneous >>>>>>>> connections. It may be OK for a performance test, but there are >>>>>>>> better >>>>>>>> ways >>>>>>>> to squeeze performance. For example, you could consider using HTTP >>>>>>>> pipelining, which is much more friendly for the server than so many >>>>>>>> connections. This is what we use when we need to send a large >>>>>>>> number >>>>>>>> of >>>>>>>> requests from other systems. There are examples of how to do that >>>>>>>> with >>>>>>>> the >>>>>>>> Python and Ruby SDKs here: >>>>>>>> >>>>>>>> Python: >>>>>>>> >>>>>>>> https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/ >>>>>>>> examples/asynchronous_inventory.py >>>>>>>> >>>>>>>> Ruby: >>>>>>>> >>>>>>>> https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/ >>>>>>>> sdk/examples/asynchronous_inventory.rb >>>>>>>> >>>>>>>> >>>>>>>> On 03/07/2018 02:43 PM, Hari Prasanth Loganathan wrote: >>>>>>>> >>>>>>>> Hi Juan, >>>>>>>> >>>>>>>> >>>>>>>> Thanks for the response. >>>>>>>>> >>>>>>>>> I agree web server can handle only limited number of concurrent >>>>>>>>> requests. >>>>>>>>> But Why it is failing with SSL handshake failure for few requests, >>>>>>>>> Can't >>>>>>>>> the JBOSS wait and serve the request? We can spare the delay but >>>>>>>>> not >>>>>>>>> with >>>>>>>>> the request fails. So Is there a configuration in oVirt which can >>>>>>>>> be >>>>>>>>> tuned >>>>>>>>> to achieve this? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Hari >>>>>>>>> >>>>>>>>> On Wed, Mar 7, 2018 at 7:05 PM, Juan Hernández < >>>>>>>>> jhernand(a)redhat.com >>>>>>>>> >>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> The first thing you will need to change for such a test is the >>>>>>>>> number >>>>>>>>> of >>>>>>>>> >>>>>>>>> simultaneous connections accepted by the Apache web server: by >>>>>>>>> default >>>>>>>>> >>>>>>>>> the >>>>>>>>>> max is 256. See the Apache documentation here: >>>>>>>>>> >>>>>>>>>> https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m >>>>>>>>>> axrequestworkers >>>>>>>>>> >>>>>>>>>> In addition I also suggest that you consider using the "worker" >>>>>>>>>> multi-processing module instead of the "prefork", as it usually >>>>>>>>>> works >>>>>>>>>> better when talking to a Java application server, because it >>>>>>>>>> re-uses >>>>>>>>>> connections better. >>>>>>>>>> >>>>>>>>>> On 03/07/2018 02:20 PM, Hari Prasanth Loganathan wrote: >>>>>>>>>> >>>>>>>>>> Hi Team, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Description of problem:* >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I am trying to achieve 1000 concurrent request to oVirt. What >>>>>>>>>>> are >>>>>>>>>>> the >>>>>>>>>>> tunable parameters to achieve this? >>>>>>>>>>> >>>>>>>>>>> I tried to perform the benchmarking for ovirt engine using >>>>>>>>>>> Apache >>>>>>>>>>> benchmark >>>>>>>>>>> using the same SSO token. >>>>>>>>>>> >>>>>>>>>>> ab -n 1000 -c 500 -k -H "accept: application/json" -H >>>>>>>>>>> "Authorization: >>>>>>>>>>> Bearer SSOTOKEN" https://172.30.56.70/ovirt-engine/ >>>>>>>>>>> <https://172.30.56.70/ovirt-engine/api/vms/5440271b-afb3-48b >>>>>>>>>>> b-9ff1-076fc07ebf50/statistics> >>>>>>>>>>> >>>>>>>>>>> When the number of concurrent request is 500, we are getting >>>>>>>>>>> more >>>>>>>>>>> than >>>>>>>>>>> 100 >>>>>>>>>>> failures with the following error, >>>>>>>>>>> >>>>>>>>>>> SSL read failed (1) - closing connection >>>>>>>>>>> 139620982339352:error: >>>>>>>>>>> >>>>>>>>>>> NOTE: It is scaling for concurrent request below 500. >>>>>>>>>>> >>>>>>>>>>> I used the profiler to get the memory and CPU and it seems very >>>>>>>>>>> less, >>>>>>>>>>> >>>>>>>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM >>>>>>>>>>> TIME+ >>>>>>>>>>> COMMAND >>>>>>>>>>> 30413 ovirt 20 0 4226664 882396 6776 S 126.0 23.0 >>>>>>>>>>> 27:48.53 >>>>>>>>>>> java >>>>>>>>>>> >>>>>>>>>>> Configuration of the machine in which Ovirt is deployed : >>>>>>>>>>> >>>>>>>>>>> RAM - 4GB, >>>>>>>>>>> Hard disk - 100GB, >>>>>>>>>>> core processor - 2, >>>>>>>>>>> OS - Cent7.x. >>>>>>>>>>> >>>>>>>>>>> In which 2GB is allocated to oVirt. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Version-Release number of selected component (if applicable): >>>>>>>>>>> >>>>>>>>>>> 4.2.2 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> How reproducible: >>>>>>>>>>> >>>>>>>>>>> If the number of concurrent requests are above 500, we are >>>>>>>>>>> easily >>>>>>>>>>> facing >>>>>>>>>>> this issue. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *Actual results:* >>>>>>>>>>> >>>>>>>>>>> SSL read failed (1) - closing connection >>>>>>>>>>> 139620982339352:error: >>>>>>>>>>> >>>>>>>>>>> *Expected results:* >>>>>>>>>>> >>>>>>>>>>> Request success. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Hari >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Users mailing list >>>>>>>>>>> Users(a)ovirt.org >>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >