Re: [ovirt-users] Tunable parameters in ovirt engine

This is the only error message we received from ab. I googled it and found that it is due to the connection drop. So It would be Great, If you could check my Apache server configuration I shared in the thread and let me know your thoughts on this. Thanks, Hari On Thu, Mar 8, 2018 at 4:56 PM, Juan Hernández <jhernand(a)redhat.com&gt; wrote: > But other than those SSL error messages, are the connections really > failing? Can you share the results reported by "ab"? > > > On 03/08/2018 12:16 PM, Hari Prasanth Loganathan wrote: > >> No Juan, It is not working with any benchmark / application tool. It fails >> with the same error SSL handshake failed (5). >> >> Could you let me know the configuration of Apache web server is correct? >> >> Thanks, >> Hari >> >> On Thu, Mar 8, 2018 at 1:08 AM, Juan Hernández <jhernand(a)redhat.com&gt; >> wrote: >> >> If you are still having problems I am inclined to think that it is a >>> client issue. For example, I'd try to remove the "-k" option from the >>> "ab" >>> command. If you use keep alive the server may decide anyhow to close the >>> connection after certain number of requests, even if the client asks to >>> keep it alive. Some clients don't handle that perfectly, "ab" may have >>> that >>> problem. If that makes the SSL error messages disappear then I think you >>> can safely ignore them, and restore the "-k" option, if you want. >>> >>> On 03/07/2018 07:30 PM, Hari Prasanth Loganathan wrote: >>> >>> Thanks Juan for your response. Appreciate it. >>>> But for some reason still, I am facing the same SSL handshake failed >>>> (5). >>>> Could you please check this configuration and let me know the issue in >>>> my >>>> ovirt engine environment. >>>> >>>> *Configuration of Apache server:* >>>> >>>> >>>> 1) httpd version, >>>> >>>> # httpd -v >>>> Server version: Apache/2.4.6 (CentOS) >>>> Server built: Oct 19 2017 20:39:16 >>>> >>>> 2) I checked the status using the following command, >>>> >>>> # systemctl status httpd -l >>>> ● httpd.service - The Apache HTTP Server >>>> Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; >>>> vendor >>>> preset: disabled) >>>> Active: active (running) since Wed 2018-03-07 23:46:32 IST; 1min >>>> 55s >>>> ago >>>> Docs: man:httpd(8) >>>> man:apachectl(8) >>>> Process: 4351 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, >>>> status=0/SUCCESS) >>>> Main PID: 4359 (httpd) >>>> Status: "Total requests: 264; Current requests/sec: 0.1; Current >>>> traffic: 204 B/sec" >>>> CGroup: /system.slice/httpd.service >>>> ├─4359 /usr/sbin/httpd -DFOREGROUND >>>> ├─4360 /usr/sbin/httpd -DFOREGROUND >>>> ├─4362 /usr/sbin/httpd -DFOREGROUND >>>> ├─5100 /usr/sbin/httpd -DFOREGROUND >>>> ├─5386 /usr/sbin/httpd -DFOREGROUND >>>> ├─5415 /usr/sbin/httpd -DFOREGROUND >>>> └─5416 /usr/sbin/httpd -DFOREGROUND >>>> >>>> 3) Since the httpd is pointing to the path : >>>> /usr/lib/systemd/system/httpd.service >>>> >>>> vi /usr/lib/systemd/system/httpd.service >>>> >>>> [Unit] >>>> Description=The Apache HTTP Server >>>> After=network.target remote-fs.target nss-lookup.target >>>> Documentation=man:httpd(8) >>>> Documentation=man:apachectl(8) >>>> >>>> [Service] >>>> Type=notify >>>> EnvironmentFile=/etc/sysconfig/httpd >>>> ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND >>>> ExecReload=/usr/sbin/httpd $OPTIONS -k graceful >>>> ExecStop=/bin/kill -WINCH ${MAINPID} >>>> # We want systemd to give httpd some time to finish gracefully, but >>>> still >>>> want >>>> # it to kill httpd after TimeoutStopSec if something went wrong during >>>> the >>>> # graceful stop. Normally, Systemd sends SIGTERM signal right after the >>>> # ExecStop, which would kill httpd. We are sending useless SIGCONT here >>>> to >>>> give >>>> # httpd time to finish. >>>> KillSignal=SIGCONT >>>> PrivateTmp=true >>>> >>>> [Install] >>>> WantedBy=multi-user.target >>>> >>>> >>>> 4) As per the above command I found the env file is available >>>> '/etc/sysconfig/httpd' >>>> >>>> vi /etc/sysconfig/httpd >>>> >>>> # >>>> # This file can be used to set additional environment variables for >>>> # the httpd process, or pass additional options to the httpd >>>> # executable. >>>> # >>>> # Note: With previous versions of httpd, the MPM could be changed by >>>> # editing an "HTTPD" variable here. With the current version, that >>>> # variable is now ignored. The MPM is a loadable module, and the >>>> # choice of MPM can be changed by editing the configuration file >>>> /etc/httpd/conf.modules.d/00-mpm.conf >>>> # >>>> >>>> # >>>> # To pass additional options (for instance, -D definitions) to the >>>> # httpd binary at startup, set OPTIONS here. >>>> # >>>> #OPTIONS= >>>> >>>> # >>>> # This setting ensures the httpd process is started in the "C" locale >>>> # by default. (Some modules will not behave correctly if >>>> # case-sensitive string comparisons are performed in a different >>>> # locale.) >>>> # >>>> LANG=C >>>> >>>> >>>> 5) As per the above command, I found that the conf fileis available in >>>> the >>>> path : /etc/httpd/conf.modules.d/00-mpm.conf >>>> >>>> vi /etc/httpd/conf.modules.d/00-mpm.conf >>>> >>>> # Select the MPM module which should be used by uncommenting exactly >>>> # one of the following LoadModule lines: >>>> >>>> # prefork MPM: Implements a non-threaded, pre-forking web server >>>> # See: http://httpd.apache.org/docs/2.4/mod/prefork.html >>>> #LoadModule mpm_prefork_module modules/mod_mpm_prefork.so >>>> >>>> # worker MPM: Multi-Processing Module implementing a hybrid >>>> # multi-threaded multi-process web server >>>> # See: http://httpd.apache.org/docs/2.4/mod/worker.html >>>> # >>>> LoadModule mpm_worker_module modules/mod_mpm_worker.so >>>> >>>> # event MPM: A variant of the worker MPM with the goal of consuming >>>> # threads only for connections with active processing >>>> # See: http://httpd.apache.org/docs/2.4/mod/event.html >>>> # >>>> #LoadModule mpm_event_module modules/mod_mpm_event.so >>>> >>>> <IfModule mpm_worker_module> >>>> ServerLimit 1000 >>>> MaxRequestWorkers 1000 >>>> </IfModule> >>>> >>>> >>>> >>>> As per your comment, I enabled the 'LoadModule mpm_worker_module >>>> modules/mod_mpm_worker.so' with the ServerLimit and MaxRequestWorkers as >>>> 1000 still I am facing the issue for the following command in apache >>>> benchmark test. >>>> >>>> Completed 100 requests >>>> SSL handshake failed (5). >>>> SSL handshake failed (5). >>>> SSL handshake failed (5). >>>> SSL handshake failed (5). >>>> SSL handshake failed (5). >>>> SSL handshake failed (5). >>>> >>>> >>>> NOTE : It always scales when I have concurrent request below 400 >>>> >>>> What is wrong in this apache configuration, why SSL handshake is failing >>>> for concurrent request above 400 ? >>>> >>>> Thanks, >>>> Hari >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Wed, Mar 7, 2018 at 9:20 PM, Juan Hernández <jhernand(a)redhat.com&gt; >>>> wrote: >>>> >>>> It means that with the default configuration the Apache web server can't >>>> >>>>> serve more than 256 concurrent connections. This applies to any >>>>> application >>>>> that uses Apache as the web frontend, not just to oVirt. If you want to >>>>> change that you have to change the MaxRequestWorkers and ServerLimit >>>>> parameters, as explained here: >>>>> >>>>> https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m >>>>> axrequestworkers >>>>> >>>>> So, go to your oVirt engine machine and create a >>>>> /etc/httpd/conf.d/my.conf >>>>> file with this content: >>>>> >>>>> MaxRequestWorkers 1000 >>>>> ServerLimit 1000 >>>>> >>>>> Then restart the Apache server: >>>>> >>>>> # systemctl restart httpd >>>>> >>>>> Then your web server should be able to handle 1000 concurrent requests, >>>>> and you will probably start to find other limits, like the amount of >>>>> memory >>>>> and CPU that those 1000 Apache child processes will consume, the number >>>>> of >>>>> threads in the JBoss application server, the number of connections to >>>>> the >>>>> database, etc. >>>>> >>>>> Let me insist a bit that if you base your benchmark solely on the >>>>> number >>>>> of concurrent requests or connections that the server can handle you >>>>> may >>>>> end up with meaningless results, as a real world application can/should >>>>> use >>>>> the server much better than that. >>>>> >>>>> On 03/07/2018 04:33 PM, Hari Prasanth Loganathan wrote: >>>>> >>>>> With the default configuration of the web server it is impossible to >>>>> >>>>>> handle >>>>>> more than 256 *connections* simultaneously. I guess that "ab" is >>>>>> opening a >>>>>> connection for each concurrent request, so when you reach request 257 >>>>>> the >>>>>> web server will just reject the connection, there is nothing that the >>>>>> JBoss >>>>>> can do about it; you have to increase the number of connections >>>>>> supported >>>>>> by the web server. >>>>>> >>>>>> *So Does it mean that oVirt cannot serve more than 257 request? * >>>>>> >>>>>> My question is, If its possible How to scale this and what is the >>>>>> configuration we need to change? >>>>>> >>>>>> Also, we are taking a benchmark in using oVirt, So I need to find the >>>>>> maximum possible oVirt request. So please let me know the >>>>>> configuration >>>>>> tuning for oVirt to achieve maximum concurrent request. >>>>>> >>>>>> Thanks, >>>>>> Hari >>>>>> >>>>>> On Wed, Mar 7, 2018 at 7:25 PM, Juan Hernández <jhernand(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>> With the default configuration of the web server it is impossible to >>>>>> >>>>>> handle more than 256 *connections* simultaneously. I guess that "ab" >>>>>>> is >>>>>>> opening a connection for each concurrent request, so when you reach >>>>>>> request >>>>>>> 257 the web server will just reject the connection, there is nothing >>>>>>> that >>>>>>> the JBoss can do about it; you have to increase the number of >>>>>>> connections >>>>>>> supported by the web server. >>>>>>> >>>>>>> Or else you may want to re-consider why you want to use 1000 >>>>>>> simultaneous >>>>>>> connections. It may be OK for a performance test, but there are >>>>>>> better >>>>>>> ways >>>>>>> to squeeze performance. For example, you could consider using HTTP >>>>>>> pipelining, which is much more friendly for the server than so many >>>>>>> connections. This is what we use when we need to send a large number >>>>>>> of >>>>>>> requests from other systems. There are examples of how to do that >>>>>>> with >>>>>>> the >>>>>>> Python and Ruby SDKs here: >>>>>>> >>>>>>> Python: >>>>>>> >>>>>>> https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/ >>>>>>> examples/asynchronous_inventory.py >>>>>>> >>>>>>> Ruby: >>>>>>> >>>>>>> https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/ >>>>>>> sdk/examples/asynchronous_inventory.rb >>>>>>> >>>>>>> >>>>>>> On 03/07/2018 02:43 PM, Hari Prasanth Loganathan wrote: >>>>>>> >>>>>>> Hi Juan, >>>>>>> >>>>>>> >>>>>>>> Thanks for the response. >>>>>>>> >>>>>>>> I agree web server can handle only limited number of concurrent >>>>>>>> requests. >>>>>>>> But Why it is failing with SSL handshake failure for few requests, >>>>>>>> Can't >>>>>>>> the JBOSS wait and serve the request? We can spare the delay but not >>>>>>>> with >>>>>>>> the request fails. So Is there a configuration in oVirt which can be >>>>>>>> tuned >>>>>>>> to achieve this? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Hari >>>>>>>> >>>>>>>> On Wed, Mar 7, 2018 at 7:05 PM, Juan Hernández <jhernand(a)redhat.com >>>>>>>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>> The first thing you will need to change for such a test is the >>>>>>>> number >>>>>>>> of >>>>>>>> >>>>>>>> simultaneous connections accepted by the Apache web server: by >>>>>>>> default >>>>>>>> >>>>>>>>> the >>>>>>>>> max is 256. See the Apache documentation here: >>>>>>>>> >>>>>>>>> https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m >>>>>>>>> axrequestworkers >>>>>>>>> >>>>>>>>> In addition I also suggest that you consider using the "worker" >>>>>>>>> multi-processing module instead of the "prefork", as it usually >>>>>>>>> works >>>>>>>>> better when talking to a Java application server, because it >>>>>>>>> re-uses >>>>>>>>> connections better. >>>>>>>>> >>>>>>>>> On 03/07/2018 02:20 PM, Hari Prasanth Loganathan wrote: >>>>>>>>> >>>>>>>>> Hi Team, >>>>>>>>> >>>>>>>>> >>>>>>>>> *Description of problem:* >>>>>>>>>> >>>>>>>>>> I am trying to achieve 1000 concurrent request to oVirt. What are >>>>>>>>>> the >>>>>>>>>> tunable parameters to achieve this? >>>>>>>>>> >>>>>>>>>> I tried to perform the benchmarking for ovirt engine using Apache >>>>>>>>>> benchmark >>>>>>>>>> using the same SSO token. >>>>>>>>>> >>>>>>>>>> ab -n 1000 -c 500 -k -H "accept: application/json" -H >>>>>>>>>> "Authorization: >>>>>>>>>> Bearer SSOTOKEN" https://172.30.56.70/ovirt-engine/ >>>>>>>>>> <https://172.30.56.70/ovirt-engine/api/vms/5440271b-afb3-48b >>>>>>>>>> b-9ff1-076fc07ebf50/statistics> >>>>>>>>>> >>>>>>>>>> When the number of concurrent request is 500, we are getting more >>>>>>>>>> than >>>>>>>>>> 100 >>>>>>>>>> failures with the following error, >>>>>>>>>> >>>>>>>>>> SSL read failed (1) - closing connection >>>>>>>>>> 139620982339352:error: >>>>>>>>>> >>>>>>>>>> NOTE: It is scaling for concurrent request below 500. >>>>>>>>>> >>>>>>>>>> I used the profiler to get the memory and CPU and it seems very >>>>>>>>>> less, >>>>>>>>>> >>>>>>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM >>>>>>>>>> TIME+ >>>>>>>>>> COMMAND >>>>>>>>>> 30413 ovirt 20 0 4226664 882396 6776 S 126.0 23.0 >>>>>>>>>> 27:48.53 >>>>>>>>>> java >>>>>>>>>> >>>>>>>>>> Configuration of the machine in which Ovirt is deployed : >>>>>>>>>> >>>>>>>>>> RAM - 4GB, >>>>>>>>>> Hard disk - 100GB, >>>>>>>>>> core processor - 2, >>>>>>>>>> OS - Cent7.x. >>>>>>>>>> >>>>>>>>>> In which 2GB is allocated to oVirt. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Version-Release number of selected component (if applicable): >>>>>>>>>> >>>>>>>>>> 4.2.2 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> How reproducible: >>>>>>>>>> >>>>>>>>>> If the number of concurrent requests are above 500, we are easily >>>>>>>>>> facing >>>>>>>>>> this issue. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Actual results:* >>>>>>>>>> >>>>>>>>>> SSL read failed (1) - closing connection >>>>>>>>>> 139620982339352:error: >>>>>>>>>> >>>>>>>>>> *Expected results:* >>>>>>>>>> >>>>>>>>>> Request success. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Hari >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Users mailing list >>>>>>>>>> Users(a)ovirt.org >>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >