Tunable parameters in ovirt engine
Hi Team, *Description of problem:* I am trying to achieve 1000 concurrent request to oVirt. What are the tunable parameters to achieve this? I tried to perform the benchmarking for ovirt engine using Apache benchmark using the same SSO token. ab -n 1000 -c 500 -k -H "accept: application/json" -H "Authorization: Bearer SSOTOKEN" https://172.30.56.70/ovirt-engine/ <https://172.30.56.70/ovirt-engine/api/vms/5440271b-afb3-48bb-9ff1-076fc07ebf50/statistics> When the number of concurrent request is 500, we are getting more than 100 failures with the following error, SSL read failed (1) - closing connection 139620982339352:error: NOTE: It is scaling for concurrent request below 500. I used the profiler to get the memory and CPU and it seems very less, PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 30413 ovirt 20 0 4226664 882396 6776 S 126.0 23.0 27:48.53 java Configuration of the machine in which Ovirt is deployed : RAM - 4GB, Hard disk - 100GB, core processor - 2, OS - Cent7.x. In which 2GB is allocated to oVirt. Version-Release number of selected component (if applicable): 4.2.2 How reproducible: If the number of concurrent requests are above 500, we are easily facing this issue. *Actual results:* SSL read failed (1) - closing connection 139620982339352:error: *Expected results:* Request success. Thanks, Hari -- DISCLAIMER The information in this e-mail is confidential and may be subject to legal privilege. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you have received this communication in error, please address with the subject heading "Received in error," send to it@msystechnologies.com, then delete the e-mail and destroy any copies of it. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. The views, opinions, conclusions and other information expressed in this electronic mail and any attachments are not given or endorsed by the company unless otherwise indicated by an authorized representative independent of this message. MSys cannot guarantee that e-mail communications are secure or error-free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses, though all reasonable precautions have been taken to ensure no viruses are present in this e-mail. As our company cannot accept responsibility for any loss or damage arising from the use of this e-mail or attachments we recommend that you subject these to your virus checking procedures prior to use
The first thing you will need to change for such a test is the number of simultaneous connections accepted by the Apache web server: by default the max is 256. See the Apache documentation here: https://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequestworkers In addition I also suggest that you consider using the "worker" multi-processing module instead of the "prefork", as it usually works better when talking to a Java application server, because it re-uses connections better. On 03/07/2018 02:20 PM, Hari Prasanth Loganathan wrote:
Hi Team,
*Description of problem:*
I am trying to achieve 1000 concurrent request to oVirt. What are the tunable parameters to achieve this?
I tried to perform the benchmarking for ovirt engine using Apache benchmark using the same SSO token.
ab -n 1000 -c 500 -k -H "accept: application/json" -H "Authorization: Bearer SSOTOKEN" https://172.30.56.70/ovirt-engine/ <https://172.30.56.70/ovirt-engine/api/vms/5440271b-afb3-48bb-9ff1-076fc07ebf50/statistics>
When the number of concurrent request is 500, we are getting more than 100 failures with the following error,
SSL read failed (1) - closing connection 139620982339352:error:
NOTE: It is scaling for concurrent request below 500.
I used the profiler to get the memory and CPU and it seems very less,
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 30413 ovirt 20 0 4226664 882396 6776 S 126.0 23.0 27:48.53 java
Configuration of the machine in which Ovirt is deployed :
RAM - 4GB, Hard disk - 100GB, core processor - 2, OS - Cent7.x.
In which 2GB is allocated to oVirt.
Version-Release number of selected component (if applicable):
4.2.2
How reproducible:
If the number of concurrent requests are above 500, we are easily facing this issue.
*Actual results:*
SSL read failed (1) - closing connection 139620982339352:error:
*Expected results:*
Request success.
Thanks, Hari
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Juan, Thanks for the response. I agree web server can handle only limited number of concurrent requests. But Why it is failing with SSL handshake failure for few requests, Can't the JBOSS wait and serve the request? We can spare the delay but not with the request fails. So Is there a configuration in oVirt which can be tuned to achieve this? Thanks, Hari On Wed, Mar 7, 2018 at 7:05 PM, Juan Hernández <jhernand@redhat.com> wrote:
The first thing you will need to change for such a test is the number of simultaneous connections accepted by the Apache web server: by default the max is 256. See the Apache documentation here:
https://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequestworkers
In addition I also suggest that you consider using the "worker" multi-processing module instead of the "prefork", as it usually works better when talking to a Java application server, because it re-uses connections better.
On 03/07/2018 02:20 PM, Hari Prasanth Loganathan wrote:
Hi Team,
*Description of problem:*
I am trying to achieve 1000 concurrent request to oVirt. What are the tunable parameters to achieve this?
I tried to perform the benchmarking for ovirt engine using Apache benchmark using the same SSO token.
ab -n 1000 -c 500 -k -H "accept: application/json" -H "Authorization: Bearer SSOTOKEN" https://172.30.56.70/ovirt-engine/ <https://172.30.56.70/ovirt-engine/api/vms/5440271b-afb3-48b b-9ff1-076fc07ebf50/statistics>
When the number of concurrent request is 500, we are getting more than 100 failures with the following error,
SSL read failed (1) - closing connection 139620982339352:error:
NOTE: It is scaling for concurrent request below 500.
I used the profiler to get the memory and CPU and it seems very less,
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 30413 ovirt 20 0 4226664 882396 6776 S 126.0 23.0 27:48.53 java
Configuration of the machine in which Ovirt is deployed :
RAM - 4GB, Hard disk - 100GB, core processor - 2, OS - Cent7.x.
In which 2GB is allocated to oVirt.
Version-Release number of selected component (if applicable):
4.2.2
How reproducible:
If the number of concurrent requests are above 500, we are easily facing this issue.
*Actual results:*
SSL read failed (1) - closing connection 139620982339352:error:
*Expected results:*
Request success.
Thanks, Hari
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- DISCLAIMER The information in this e-mail is confidential and may be subject to legal privilege. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you have received this communication in error, please address with the subject heading "Received in error," send to it@msystechnologies.com, then delete the e-mail and destroy any copies of it. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. The views, opinions, conclusions and other information expressed in this electronic mail and any attachments are not given or endorsed by the company unless otherwise indicated by an authorized representative independent of this message. MSys cannot guarantee that e-mail communications are secure or error-free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses, though all reasonable precautions have been taken to ensure no viruses are present in this e-mail. As our company cannot accept responsibility for any loss or damage arising from the use of this e-mail or attachments we recommend that you subject these to your virus checking procedures prior to use
With the default configuration of the web server it is impossible to handle more than 256 *connections* simultaneously. I guess that "ab" is opening a connection for each concurrent request, so when you reach request 257 the web server will just reject the connection, there is nothing that the JBoss can do about it; you have to increase the number of connections supported by the web server. Or else you may want to re-consider why you want to use 1000 simultaneous connections. It may be OK for a performance test, but there are better ways to squeeze performance. For example, you could consider using HTTP pipelining, which is much more friendly for the server than so many connections. This is what we use when we need to send a large number of requests from other systems. There are examples of how to do that with the Python and Ruby SDKs here: Python: https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/asynchron... Ruby: https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/sdk/examples/asyn... On 03/07/2018 02:43 PM, Hari Prasanth Loganathan wrote:
Hi Juan,
Thanks for the response.
I agree web server can handle only limited number of concurrent requests. But Why it is failing with SSL handshake failure for few requests, Can't the JBOSS wait and serve the request? We can spare the delay but not with the request fails. So Is there a configuration in oVirt which can be tuned to achieve this?
Thanks, Hari
On Wed, Mar 7, 2018 at 7:05 PM, Juan Hernández <jhernand@redhat.com> wrote:
The first thing you will need to change for such a test is the number of simultaneous connections accepted by the Apache web server: by default the max is 256. See the Apache documentation here:
https://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequestworkers
In addition I also suggest that you consider using the "worker" multi-processing module instead of the "prefork", as it usually works better when talking to a Java application server, because it re-uses connections better.
On 03/07/2018 02:20 PM, Hari Prasanth Loganathan wrote:
Hi Team,
*Description of problem:*
I am trying to achieve 1000 concurrent request to oVirt. What are the tunable parameters to achieve this?
I tried to perform the benchmarking for ovirt engine using Apache benchmark using the same SSO token.
ab -n 1000 -c 500 -k -H "accept: application/json" -H "Authorization: Bearer SSOTOKEN" https://172.30.56.70/ovirt-engine/ <https://172.30.56.70/ovirt-engine/api/vms/5440271b-afb3-48b b-9ff1-076fc07ebf50/statistics>
When the number of concurrent request is 500, we are getting more than 100 failures with the following error,
SSL read failed (1) - closing connection 139620982339352:error:
NOTE: It is scaling for concurrent request below 500.
I used the profiler to get the memory and CPU and it seems very less,
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 30413 ovirt 20 0 4226664 882396 6776 S 126.0 23.0 27:48.53 java
Configuration of the machine in which Ovirt is deployed :
RAM - 4GB, Hard disk - 100GB, core processor - 2, OS - Cent7.x.
In which 2GB is allocated to oVirt.
Version-Release number of selected component (if applicable):
4.2.2
How reproducible:
If the number of concurrent requests are above 500, we are easily facing this issue.
*Actual results:*
SSL read failed (1) - closing connection 139620982339352:error:
*Expected results:*
Request success.
Thanks, Hari
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
With the default configuration of the web server it is impossible to handle more than 256 *connections* simultaneously. I guess that "ab" is opening a connection for each concurrent request, so when you reach request 257 the web server will just reject the connection, there is nothing that the JBoss can do about it; you have to increase the number of connections supported by the web server. *So Does it mean that oVirt cannot serve more than 257 request? * My question is, If its possible How to scale this and what is the configuration we need to change? Also, we are taking a benchmark in using oVirt, So I need to find the maximum possible oVirt request. So please let me know the configuration tuning for oVirt to achieve maximum concurrent request. Thanks, Hari On Wed, Mar 7, 2018 at 7:25 PM, Juan Hernández <jhernand@redhat.com> wrote:
With the default configuration of the web server it is impossible to handle more than 256 *connections* simultaneously. I guess that "ab" is opening a connection for each concurrent request, so when you reach request 257 the web server will just reject the connection, there is nothing that the JBoss can do about it; you have to increase the number of connections supported by the web server.
Or else you may want to re-consider why you want to use 1000 simultaneous connections. It may be OK for a performance test, but there are better ways to squeeze performance. For example, you could consider using HTTP pipelining, which is much more friendly for the server than so many connections. This is what we use when we need to send a large number of requests from other systems. There are examples of how to do that with the Python and Ruby SDKs here:
Python:
https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/ examples/asynchronous_inventory.py
Ruby:
https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/ sdk/examples/asynchronous_inventory.rb
On 03/07/2018 02:43 PM, Hari Prasanth Loganathan wrote:
Hi Juan,
Thanks for the response.
I agree web server can handle only limited number of concurrent requests. But Why it is failing with SSL handshake failure for few requests, Can't the JBOSS wait and serve the request? We can spare the delay but not with the request fails. So Is there a configuration in oVirt which can be tuned to achieve this?
Thanks, Hari
On Wed, Mar 7, 2018 at 7:05 PM, Juan Hernández <jhernand@redhat.com> wrote:
The first thing you will need to change for such a test is the number of
simultaneous connections accepted by the Apache web server: by default the max is 256. See the Apache documentation here:
https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m axrequestworkers
In addition I also suggest that you consider using the "worker" multi-processing module instead of the "prefork", as it usually works better when talking to a Java application server, because it re-uses connections better.
On 03/07/2018 02:20 PM, Hari Prasanth Loganathan wrote:
Hi Team,
*Description of problem:*
I am trying to achieve 1000 concurrent request to oVirt. What are the tunable parameters to achieve this?
I tried to perform the benchmarking for ovirt engine using Apache benchmark using the same SSO token.
ab -n 1000 -c 500 -k -H "accept: application/json" -H "Authorization: Bearer SSOTOKEN" https://172.30.56.70/ovirt-engine/ <https://172.30.56.70/ovirt-engine/api/vms/5440271b-afb3-48b b-9ff1-076fc07ebf50/statistics>
When the number of concurrent request is 500, we are getting more than 100 failures with the following error,
SSL read failed (1) - closing connection 139620982339352:error:
NOTE: It is scaling for concurrent request below 500.
I used the profiler to get the memory and CPU and it seems very less,
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 30413 ovirt 20 0 4226664 882396 6776 S 126.0 23.0 27:48.53 java
Configuration of the machine in which Ovirt is deployed :
RAM - 4GB, Hard disk - 100GB, core processor - 2, OS - Cent7.x.
In which 2GB is allocated to oVirt.
Version-Release number of selected component (if applicable):
4.2.2
How reproducible:
If the number of concurrent requests are above 500, we are easily facing this issue.
*Actual results:*
SSL read failed (1) - closing connection 139620982339352:error:
*Expected results:*
Request success.
Thanks, Hari
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- DISCLAIMER The information in this e-mail is confidential and may be subject to legal privilege. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you have received this communication in error, please address with the subject heading "Received in error," send to it@msystechnologies.com, then delete the e-mail and destroy any copies of it. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. The views, opinions, conclusions and other information expressed in this electronic mail and any attachments are not given or endorsed by the company unless otherwise indicated by an authorized representative independent of this message. MSys cannot guarantee that e-mail communications are secure or error-free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses, though all reasonable precautions have been taken to ensure no viruses are present in this e-mail. As our company cannot accept responsibility for any loss or damage arising from the use of this e-mail or attachments we recommend that you subject these to your virus checking procedures prior to use
It means that with the default configuration the Apache web server can't serve more than 256 concurrent connections. This applies to any application that uses Apache as the web frontend, not just to oVirt. If you want to change that you have to change the MaxRequestWorkers and ServerLimit parameters, as explained here: https://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequestworkers So, go to your oVirt engine machine and create a /etc/httpd/conf.d/my.conf file with this content: MaxRequestWorkers 1000 ServerLimit 1000 Then restart the Apache server: # systemctl restart httpd Then your web server should be able to handle 1000 concurrent requests, and you will probably start to find other limits, like the amount of memory and CPU that those 1000 Apache child processes will consume, the number of threads in the JBoss application server, the number of connections to the database, etc. Let me insist a bit that if you base your benchmark solely on the number of concurrent requests or connections that the server can handle you may end up with meaningless results, as a real world application can/should use the server much better than that. On 03/07/2018 04:33 PM, Hari Prasanth Loganathan wrote:
With the default configuration of the web server it is impossible to handle more than 256 *connections* simultaneously. I guess that "ab" is opening a connection for each concurrent request, so when you reach request 257 the web server will just reject the connection, there is nothing that the JBoss can do about it; you have to increase the number of connections supported by the web server.
*So Does it mean that oVirt cannot serve more than 257 request? * My question is, If its possible How to scale this and what is the configuration we need to change?
Also, we are taking a benchmark in using oVirt, So I need to find the maximum possible oVirt request. So please let me know the configuration tuning for oVirt to achieve maximum concurrent request.
Thanks, Hari
On Wed, Mar 7, 2018 at 7:25 PM, Juan Hernández <jhernand@redhat.com> wrote:
With the default configuration of the web server it is impossible to handle more than 256 *connections* simultaneously. I guess that "ab" is opening a connection for each concurrent request, so when you reach request 257 the web server will just reject the connection, there is nothing that the JBoss can do about it; you have to increase the number of connections supported by the web server.
Or else you may want to re-consider why you want to use 1000 simultaneous connections. It may be OK for a performance test, but there are better ways to squeeze performance. For example, you could consider using HTTP pipelining, which is much more friendly for the server than so many connections. This is what we use when we need to send a large number of requests from other systems. There are examples of how to do that with the Python and Ruby SDKs here:
Python:
https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/ examples/asynchronous_inventory.py
Ruby:
https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/ sdk/examples/asynchronous_inventory.rb
On 03/07/2018 02:43 PM, Hari Prasanth Loganathan wrote:
Hi Juan,
Thanks for the response.
I agree web server can handle only limited number of concurrent requests. But Why it is failing with SSL handshake failure for few requests, Can't the JBOSS wait and serve the request? We can spare the delay but not with the request fails. So Is there a configuration in oVirt which can be tuned to achieve this?
Thanks, Hari
On Wed, Mar 7, 2018 at 7:05 PM, Juan Hernández <jhernand@redhat.com> wrote:
The first thing you will need to change for such a test is the number of
simultaneous connections accepted by the Apache web server: by default the max is 256. See the Apache documentation here:
https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m axrequestworkers
In addition I also suggest that you consider using the "worker" multi-processing module instead of the "prefork", as it usually works better when talking to a Java application server, because it re-uses connections better.
On 03/07/2018 02:20 PM, Hari Prasanth Loganathan wrote:
Hi Team,
*Description of problem:*
I am trying to achieve 1000 concurrent request to oVirt. What are the tunable parameters to achieve this?
I tried to perform the benchmarking for ovirt engine using Apache benchmark using the same SSO token.
ab -n 1000 -c 500 -k -H "accept: application/json" -H "Authorization: Bearer SSOTOKEN" https://172.30.56.70/ovirt-engine/ <https://172.30.56.70/ovirt-engine/api/vms/5440271b-afb3-48b b-9ff1-076fc07ebf50/statistics>
When the number of concurrent request is 500, we are getting more than 100 failures with the following error,
SSL read failed (1) - closing connection 139620982339352:error:
NOTE: It is scaling for concurrent request below 500.
I used the profiler to get the memory and CPU and it seems very less,
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 30413 ovirt 20 0 4226664 882396 6776 S 126.0 23.0 27:48.53 java
Configuration of the machine in which Ovirt is deployed :
RAM - 4GB, Hard disk - 100GB, core processor - 2, OS - Cent7.x.
In which 2GB is allocated to oVirt.
Version-Release number of selected component (if applicable):
4.2.2
How reproducible:
If the number of concurrent requests are above 500, we are easily facing this issue.
*Actual results:*
SSL read failed (1) - closing connection 139620982339352:error:
*Expected results:*
Request success.
Thanks, Hari
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Hari, any specific use-case you are trying to achieve with this benchmark? what are you trying to simulate? just wondering whether there are different options to achieve it.. Thanks. On Wed, Mar 7, 2018 at 5:50 PM, Juan Hernández <jhernand@redhat.com> wrote:
It means that with the default configuration the Apache web server can't serve more than 256 concurrent connections. This applies to any application that uses Apache as the web frontend, not just to oVirt. If you want to change that you have to change the MaxRequestWorkers and ServerLimit parameters, as explained here:
https://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequestworkers
So, go to your oVirt engine machine and create a /etc/httpd/conf.d/my.conf file with this content:
MaxRequestWorkers 1000 ServerLimit 1000
Then restart the Apache server:
# systemctl restart httpd
Then your web server should be able to handle 1000 concurrent requests, and you will probably start to find other limits, like the amount of memory and CPU that those 1000 Apache child processes will consume, the number of threads in the JBoss application server, the number of connections to the database, etc.
Let me insist a bit that if you base your benchmark solely on the number of concurrent requests or connections that the server can handle you may end up with meaningless results, as a real world application can/should use the server much better than that.
On 03/07/2018 04:33 PM, Hari Prasanth Loganathan wrote:
With the default configuration of the web server it is impossible to handle more than 256 *connections* simultaneously. I guess that "ab" is opening a connection for each concurrent request, so when you reach request 257 the web server will just reject the connection, there is nothing that the JBoss can do about it; you have to increase the number of connections supported by the web server.
*So Does it mean that oVirt cannot serve more than 257 request? *
My question is, If its possible How to scale this and what is the configuration we need to change?
Also, we are taking a benchmark in using oVirt, So I need to find the maximum possible oVirt request. So please let me know the configuration tuning for oVirt to achieve maximum concurrent request.
Thanks, Hari
On Wed, Mar 7, 2018 at 7:25 PM, Juan Hernández <jhernand@redhat.com> wrote:
With the default configuration of the web server it is impossible to
handle more than 256 *connections* simultaneously. I guess that "ab" is opening a connection for each concurrent request, so when you reach request 257 the web server will just reject the connection, there is nothing that the JBoss can do about it; you have to increase the number of connections supported by the web server.
Or else you may want to re-consider why you want to use 1000 simultaneous connections. It may be OK for a performance test, but there are better ways to squeeze performance. For example, you could consider using HTTP pipelining, which is much more friendly for the server than so many connections. This is what we use when we need to send a large number of requests from other systems. There are examples of how to do that with the Python and Ruby SDKs here:
Python:
https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/ examples/asynchronous_inventory.py
Ruby:
https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/ sdk/examples/asynchronous_inventory.rb
On 03/07/2018 02:43 PM, Hari Prasanth Loganathan wrote:
Hi Juan,
Thanks for the response.
I agree web server can handle only limited number of concurrent requests. But Why it is failing with SSL handshake failure for few requests, Can't the JBOSS wait and serve the request? We can spare the delay but not with the request fails. So Is there a configuration in oVirt which can be tuned to achieve this?
Thanks, Hari
On Wed, Mar 7, 2018 at 7:05 PM, Juan Hernández <jhernand@redhat.com> wrote:
The first thing you will need to change for such a test is the number of
simultaneous connections accepted by the Apache web server: by default the max is 256. See the Apache documentation here:
https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m axrequestworkers
In addition I also suggest that you consider using the "worker" multi-processing module instead of the "prefork", as it usually works better when talking to a Java application server, because it re-uses connections better.
On 03/07/2018 02:20 PM, Hari Prasanth Loganathan wrote:
Hi Team,
*Description of problem:*
I am trying to achieve 1000 concurrent request to oVirt. What are the tunable parameters to achieve this?
I tried to perform the benchmarking for ovirt engine using Apache benchmark using the same SSO token.
ab -n 1000 -c 500 -k -H "accept: application/json" -H "Authorization: Bearer SSOTOKEN" https://172.30.56.70/ovirt-engine/ <https://172.30.56.70/ovirt-engine/api/vms/5440271b-afb3-48b b-9ff1-076fc07ebf50/statistics>
When the number of concurrent request is 500, we are getting more than 100 failures with the following error,
SSL read failed (1) - closing connection 139620982339352:error:
NOTE: It is scaling for concurrent request below 500.
I used the profiler to get the memory and CPU and it seems very less,
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 30413 ovirt 20 0 4226664 882396 6776 S 126.0 23.0 27:48.53 java
Configuration of the machine in which Ovirt is deployed :
RAM - 4GB, Hard disk - 100GB, core processor - 2, OS - Cent7.x.
In which 2GB is allocated to oVirt.
Version-Release number of selected component (if applicable):
4.2.2
How reproducible:
If the number of concurrent requests are above 500, we are easily facing this issue.
*Actual results:*
SSL read failed (1) - closing connection 139620982339352:error:
*Expected results:*
Request success.
Thanks, Hari
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Thanks Juan for your response. Appreciate it. But for some reason still, I am facing the same SSL handshake failed (5). Could you please check this configuration and let me know the issue in my ovirt engine environment. *Configuration of Apache server:* 1) httpd version, # httpd -v Server version: Apache/2.4.6 (CentOS) Server built: Oct 19 2017 20:39:16 2) I checked the status using the following command, # systemctl status httpd -l ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2018-03-07 23:46:32 IST; 1min 55s ago Docs: man:httpd(8) man:apachectl(8) Process: 4351 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS) Main PID: 4359 (httpd) Status: "Total requests: 264; Current requests/sec: 0.1; Current traffic: 204 B/sec" CGroup: /system.slice/httpd.service ├─4359 /usr/sbin/httpd -DFOREGROUND ├─4360 /usr/sbin/httpd -DFOREGROUND ├─4362 /usr/sbin/httpd -DFOREGROUND ├─5100 /usr/sbin/httpd -DFOREGROUND ├─5386 /usr/sbin/httpd -DFOREGROUND ├─5415 /usr/sbin/httpd -DFOREGROUND └─5416 /usr/sbin/httpd -DFOREGROUND 3) Since the httpd is pointing to the path : /usr/lib/systemd/system/httpd.service vi /usr/lib/systemd/system/httpd.service [Unit] Description=The Apache HTTP Server After=network.target remote-fs.target nss-lookup.target Documentation=man:httpd(8) Documentation=man:apachectl(8) [Service] Type=notify EnvironmentFile=/etc/sysconfig/httpd ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND ExecReload=/usr/sbin/httpd $OPTIONS -k graceful ExecStop=/bin/kill -WINCH ${MAINPID} # We want systemd to give httpd some time to finish gracefully, but still want # it to kill httpd after TimeoutStopSec if something went wrong during the # graceful stop. Normally, Systemd sends SIGTERM signal right after the # ExecStop, which would kill httpd. We are sending useless SIGCONT here to give # httpd time to finish. KillSignal=SIGCONT PrivateTmp=true [Install] WantedBy=multi-user.target 4) As per the above command I found the env file is available '/etc/sysconfig/httpd' vi /etc/sysconfig/httpd # # This file can be used to set additional environment variables for # the httpd process, or pass additional options to the httpd # executable. # # Note: With previous versions of httpd, the MPM could be changed by # editing an "HTTPD" variable here. With the current version, that # variable is now ignored. The MPM is a loadable module, and the # choice of MPM can be changed by editing the configuration file /etc/httpd/conf.modules.d/00-mpm.conf # # # To pass additional options (for instance, -D definitions) to the # httpd binary at startup, set OPTIONS here. # #OPTIONS= # # This setting ensures the httpd process is started in the "C" locale # by default. (Some modules will not behave correctly if # case-sensitive string comparisons are performed in a different # locale.) # LANG=C 5) As per the above command, I found that the conf fileis available in the path : /etc/httpd/conf.modules.d/00-mpm.conf vi /etc/httpd/conf.modules.d/00-mpm.conf # Select the MPM module which should be used by uncommenting exactly # one of the following LoadModule lines: # prefork MPM: Implements a non-threaded, pre-forking web server # See: http://httpd.apache.org/docs/2.4/mod/prefork.html #LoadModule mpm_prefork_module modules/mod_mpm_prefork.so # worker MPM: Multi-Processing Module implementing a hybrid # multi-threaded multi-process web server # See: http://httpd.apache.org/docs/2.4/mod/worker.html # LoadModule mpm_worker_module modules/mod_mpm_worker.so # event MPM: A variant of the worker MPM with the goal of consuming # threads only for connections with active processing # See: http://httpd.apache.org/docs/2.4/mod/event.html # #LoadModule mpm_event_module modules/mod_mpm_event.so <IfModule mpm_worker_module> ServerLimit 1000 MaxRequestWorkers 1000 </IfModule> As per your comment, I enabled the 'LoadModule mpm_worker_module modules/mod_mpm_worker.so' with the ServerLimit and MaxRequestWorkers as 1000 still I am facing the issue for the following command in apache benchmark test. Completed 100 requests SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). NOTE : It always scales when I have concurrent request below 400 What is wrong in this apache configuration, why SSL handshake is failing for concurrent request above 400 ? Thanks, Hari On Wed, Mar 7, 2018 at 9:20 PM, Juan Hernández <jhernand@redhat.com> wrote:
It means that with the default configuration the Apache web server can't serve more than 256 concurrent connections. This applies to any application that uses Apache as the web frontend, not just to oVirt. If you want to change that you have to change the MaxRequestWorkers and ServerLimit parameters, as explained here:
https://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequestworkers
So, go to your oVirt engine machine and create a /etc/httpd/conf.d/my.conf file with this content:
MaxRequestWorkers 1000 ServerLimit 1000
Then restart the Apache server:
# systemctl restart httpd
Then your web server should be able to handle 1000 concurrent requests, and you will probably start to find other limits, like the amount of memory and CPU that those 1000 Apache child processes will consume, the number of threads in the JBoss application server, the number of connections to the database, etc.
Let me insist a bit that if you base your benchmark solely on the number of concurrent requests or connections that the server can handle you may end up with meaningless results, as a real world application can/should use the server much better than that.
On 03/07/2018 04:33 PM, Hari Prasanth Loganathan wrote:
With the default configuration of the web server it is impossible to handle more than 256 *connections* simultaneously. I guess that "ab" is opening a connection for each concurrent request, so when you reach request 257 the web server will just reject the connection, there is nothing that the JBoss can do about it; you have to increase the number of connections supported by the web server.
*So Does it mean that oVirt cannot serve more than 257 request? *
My question is, If its possible How to scale this and what is the configuration we need to change?
Also, we are taking a benchmark in using oVirt, So I need to find the maximum possible oVirt request. So please let me know the configuration tuning for oVirt to achieve maximum concurrent request.
Thanks, Hari
On Wed, Mar 7, 2018 at 7:25 PM, Juan Hernández <jhernand@redhat.com> wrote:
With the default configuration of the web server it is impossible to
handle more than 256 *connections* simultaneously. I guess that "ab" is opening a connection for each concurrent request, so when you reach request 257 the web server will just reject the connection, there is nothing that the JBoss can do about it; you have to increase the number of connections supported by the web server.
Or else you may want to re-consider why you want to use 1000 simultaneous connections. It may be OK for a performance test, but there are better ways to squeeze performance. For example, you could consider using HTTP pipelining, which is much more friendly for the server than so many connections. This is what we use when we need to send a large number of requests from other systems. There are examples of how to do that with the Python and Ruby SDKs here:
Python:
https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/ examples/asynchronous_inventory.py
Ruby:
https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/ sdk/examples/asynchronous_inventory.rb
On 03/07/2018 02:43 PM, Hari Prasanth Loganathan wrote:
Hi Juan,
Thanks for the response.
I agree web server can handle only limited number of concurrent requests. But Why it is failing with SSL handshake failure for few requests, Can't the JBOSS wait and serve the request? We can spare the delay but not with the request fails. So Is there a configuration in oVirt which can be tuned to achieve this?
Thanks, Hari
On Wed, Mar 7, 2018 at 7:05 PM, Juan Hernández <jhernand@redhat.com> wrote:
The first thing you will need to change for such a test is the number of
simultaneous connections accepted by the Apache web server: by default the max is 256. See the Apache documentation here:
https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m axrequestworkers
In addition I also suggest that you consider using the "worker" multi-processing module instead of the "prefork", as it usually works better when talking to a Java application server, because it re-uses connections better.
On 03/07/2018 02:20 PM, Hari Prasanth Loganathan wrote:
Hi Team,
*Description of problem:*
I am trying to achieve 1000 concurrent request to oVirt. What are the tunable parameters to achieve this?
I tried to perform the benchmarking for ovirt engine using Apache benchmark using the same SSO token.
ab -n 1000 -c 500 -k -H "accept: application/json" -H "Authorization: Bearer SSOTOKEN" https://172.30.56.70/ovirt-engine/ <https://172.30.56.70/ovirt-engine/api/vms/5440271b-afb3-48b b-9ff1-076fc07ebf50/statistics>
When the number of concurrent request is 500, we are getting more than 100 failures with the following error,
SSL read failed (1) - closing connection 139620982339352:error:
NOTE: It is scaling for concurrent request below 500.
I used the profiler to get the memory and CPU and it seems very less,
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 30413 ovirt 20 0 4226664 882396 6776 S 126.0 23.0 27:48.53 java
Configuration of the machine in which Ovirt is deployed :
RAM - 4GB, Hard disk - 100GB, core processor - 2, OS - Cent7.x.
In which 2GB is allocated to oVirt.
Version-Release number of selected component (if applicable):
4.2.2
How reproducible:
If the number of concurrent requests are above 500, we are easily facing this issue.
*Actual results:*
SSL read failed (1) - closing connection 139620982339352:error:
*Expected results:*
Request success.
Thanks, Hari
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- DISCLAIMER The information in this e-mail is confidential and may be subject to legal privilege. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you have received this communication in error, please address with the subject heading "Received in error," send to it@msystechnologies.com, then delete the e-mail and destroy any copies of it. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. The views, opinions, conclusions and other information expressed in this electronic mail and any attachments are not given or endorsed by the company unless otherwise indicated by an authorized representative independent of this message. MSys cannot guarantee that e-mail communications are secure or error-free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses, though all reasonable precautions have been taken to ensure no viruses are present in this e-mail. As our company cannot accept responsibility for any loss or damage arising from the use of this e-mail or attachments we recommend that you subject these to your virus checking procedures prior to use
If you are still having problems I am inclined to think that it is a client issue. For example, I'd try to remove the "-k" option from the "ab" command. If you use keep alive the server may decide anyhow to close the connection after certain number of requests, even if the client asks to keep it alive. Some clients don't handle that perfectly, "ab" may have that problem. If that makes the SSL error messages disappear then I think you can safely ignore them, and restore the "-k" option, if you want. On 03/07/2018 07:30 PM, Hari Prasanth Loganathan wrote:
Thanks Juan for your response. Appreciate it. But for some reason still, I am facing the same SSL handshake failed (5). Could you please check this configuration and let me know the issue in my ovirt engine environment.
*Configuration of Apache server:*
1) httpd version,
# httpd -v Server version: Apache/2.4.6 (CentOS) Server built: Oct 19 2017 20:39:16
2) I checked the status using the following command,
# systemctl status httpd -l ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2018-03-07 23:46:32 IST; 1min 55s ago Docs: man:httpd(8) man:apachectl(8) Process: 4351 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS) Main PID: 4359 (httpd) Status: "Total requests: 264; Current requests/sec: 0.1; Current traffic: 204 B/sec" CGroup: /system.slice/httpd.service ├─4359 /usr/sbin/httpd -DFOREGROUND ├─4360 /usr/sbin/httpd -DFOREGROUND ├─4362 /usr/sbin/httpd -DFOREGROUND ├─5100 /usr/sbin/httpd -DFOREGROUND ├─5386 /usr/sbin/httpd -DFOREGROUND ├─5415 /usr/sbin/httpd -DFOREGROUND └─5416 /usr/sbin/httpd -DFOREGROUND
3) Since the httpd is pointing to the path : /usr/lib/systemd/system/httpd.service
vi /usr/lib/systemd/system/httpd.service
[Unit] Description=The Apache HTTP Server After=network.target remote-fs.target nss-lookup.target Documentation=man:httpd(8) Documentation=man:apachectl(8)
[Service] Type=notify EnvironmentFile=/etc/sysconfig/httpd ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND ExecReload=/usr/sbin/httpd $OPTIONS -k graceful ExecStop=/bin/kill -WINCH ${MAINPID} # We want systemd to give httpd some time to finish gracefully, but still want # it to kill httpd after TimeoutStopSec if something went wrong during the # graceful stop. Normally, Systemd sends SIGTERM signal right after the # ExecStop, which would kill httpd. We are sending useless SIGCONT here to give # httpd time to finish. KillSignal=SIGCONT PrivateTmp=true
[Install] WantedBy=multi-user.target
4) As per the above command I found the env file is available '/etc/sysconfig/httpd'
vi /etc/sysconfig/httpd
# # This file can be used to set additional environment variables for # the httpd process, or pass additional options to the httpd # executable. # # Note: With previous versions of httpd, the MPM could be changed by # editing an "HTTPD" variable here. With the current version, that # variable is now ignored. The MPM is a loadable module, and the # choice of MPM can be changed by editing the configuration file /etc/httpd/conf.modules.d/00-mpm.conf #
# # To pass additional options (for instance, -D definitions) to the # httpd binary at startup, set OPTIONS here. # #OPTIONS=
# # This setting ensures the httpd process is started in the "C" locale # by default. (Some modules will not behave correctly if # case-sensitive string comparisons are performed in a different # locale.) # LANG=C
5) As per the above command, I found that the conf fileis available in the path : /etc/httpd/conf.modules.d/00-mpm.conf
vi /etc/httpd/conf.modules.d/00-mpm.conf
# Select the MPM module which should be used by uncommenting exactly # one of the following LoadModule lines:
# prefork MPM: Implements a non-threaded, pre-forking web server # See: http://httpd.apache.org/docs/2.4/mod/prefork.html #LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
# worker MPM: Multi-Processing Module implementing a hybrid # multi-threaded multi-process web server # See: http://httpd.apache.org/docs/2.4/mod/worker.html # LoadModule mpm_worker_module modules/mod_mpm_worker.so
# event MPM: A variant of the worker MPM with the goal of consuming # threads only for connections with active processing # See: http://httpd.apache.org/docs/2.4/mod/event.html # #LoadModule mpm_event_module modules/mod_mpm_event.so
<IfModule mpm_worker_module> ServerLimit 1000 MaxRequestWorkers 1000 </IfModule>
As per your comment, I enabled the 'LoadModule mpm_worker_module modules/mod_mpm_worker.so' with the ServerLimit and MaxRequestWorkers as 1000 still I am facing the issue for the following command in apache benchmark test.
Completed 100 requests SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5).
NOTE : It always scales when I have concurrent request below 400
What is wrong in this apache configuration, why SSL handshake is failing for concurrent request above 400 ?
Thanks, Hari
On Wed, Mar 7, 2018 at 9:20 PM, Juan Hernández <jhernand@redhat.com> wrote:
It means that with the default configuration the Apache web server can't serve more than 256 concurrent connections. This applies to any application that uses Apache as the web frontend, not just to oVirt. If you want to change that you have to change the MaxRequestWorkers and ServerLimit parameters, as explained here:
https://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequestworkers
So, go to your oVirt engine machine and create a /etc/httpd/conf.d/my.conf file with this content:
MaxRequestWorkers 1000 ServerLimit 1000
Then restart the Apache server:
# systemctl restart httpd
Then your web server should be able to handle 1000 concurrent requests, and you will probably start to find other limits, like the amount of memory and CPU that those 1000 Apache child processes will consume, the number of threads in the JBoss application server, the number of connections to the database, etc.
Let me insist a bit that if you base your benchmark solely on the number of concurrent requests or connections that the server can handle you may end up with meaningless results, as a real world application can/should use the server much better than that.
On 03/07/2018 04:33 PM, Hari Prasanth Loganathan wrote:
With the default configuration of the web server it is impossible to handle more than 256 *connections* simultaneously. I guess that "ab" is opening a connection for each concurrent request, so when you reach request 257 the web server will just reject the connection, there is nothing that the JBoss can do about it; you have to increase the number of connections supported by the web server.
*So Does it mean that oVirt cannot serve more than 257 request? *
My question is, If its possible How to scale this and what is the configuration we need to change?
Also, we are taking a benchmark in using oVirt, So I need to find the maximum possible oVirt request. So please let me know the configuration tuning for oVirt to achieve maximum concurrent request.
Thanks, Hari
On Wed, Mar 7, 2018 at 7:25 PM, Juan Hernández <jhernand@redhat.com> wrote:
With the default configuration of the web server it is impossible to
handle more than 256 *connections* simultaneously. I guess that "ab" is opening a connection for each concurrent request, so when you reach request 257 the web server will just reject the connection, there is nothing that the JBoss can do about it; you have to increase the number of connections supported by the web server.
Or else you may want to re-consider why you want to use 1000 simultaneous connections. It may be OK for a performance test, but there are better ways to squeeze performance. For example, you could consider using HTTP pipelining, which is much more friendly for the server than so many connections. This is what we use when we need to send a large number of requests from other systems. There are examples of how to do that with the Python and Ruby SDKs here:
Python:
https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/ examples/asynchronous_inventory.py
Ruby:
https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/ sdk/examples/asynchronous_inventory.rb
On 03/07/2018 02:43 PM, Hari Prasanth Loganathan wrote:
Hi Juan,
Thanks for the response.
I agree web server can handle only limited number of concurrent requests. But Why it is failing with SSL handshake failure for few requests, Can't the JBOSS wait and serve the request? We can spare the delay but not with the request fails. So Is there a configuration in oVirt which can be tuned to achieve this?
Thanks, Hari
On Wed, Mar 7, 2018 at 7:05 PM, Juan Hernández <jhernand@redhat.com> wrote:
The first thing you will need to change for such a test is the number of
simultaneous connections accepted by the Apache web server: by default the max is 256. See the Apache documentation here:
https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m axrequestworkers
In addition I also suggest that you consider using the "worker" multi-processing module instead of the "prefork", as it usually works better when talking to a Java application server, because it re-uses connections better.
On 03/07/2018 02:20 PM, Hari Prasanth Loganathan wrote:
Hi Team,
> > *Description of problem:* > > I am trying to achieve 1000 concurrent request to oVirt. What are the > tunable parameters to achieve this? > > I tried to perform the benchmarking for ovirt engine using Apache > benchmark > using the same SSO token. > > ab -n 1000 -c 500 -k -H "accept: application/json" -H "Authorization: > Bearer SSOTOKEN" https://172.30.56.70/ovirt-engine/ > <https://172.30.56.70/ovirt-engine/api/vms/5440271b-afb3-48b > b-9ff1-076fc07ebf50/statistics> > > When the number of concurrent request is 500, we are getting more than > 100 > failures with the following error, > > SSL read failed (1) - closing connection > 139620982339352:error: > > NOTE: It is scaling for concurrent request below 500. > > I used the profiler to get the memory and CPU and it seems very less, > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 30413 ovirt 20 0 4226664 882396 6776 S 126.0 23.0 27:48.53 > java > > Configuration of the machine in which Ovirt is deployed : > > RAM - 4GB, > Hard disk - 100GB, > core processor - 2, > OS - Cent7.x. > > In which 2GB is allocated to oVirt. > > > Version-Release number of selected component (if applicable): > > 4.2.2 > > > How reproducible: > > If the number of concurrent requests are above 500, we are easily > facing > this issue. > > > *Actual results:* > > SSL read failed (1) - closing connection > 139620982339352:error: > > *Expected results:* > > Request success. > > > Thanks, > Hari > > > > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > > >
No Juan, It is not working with any benchmark / application tool. It fails with the same error SSL handshake failed (5). Could you let me know the configuration of Apache web server is correct? Thanks, Hari On Thu, Mar 8, 2018 at 1:08 AM, Juan Hernández <jhernand@redhat.com> wrote:
If you are still having problems I am inclined to think that it is a client issue. For example, I'd try to remove the "-k" option from the "ab" command. If you use keep alive the server may decide anyhow to close the connection after certain number of requests, even if the client asks to keep it alive. Some clients don't handle that perfectly, "ab" may have that problem. If that makes the SSL error messages disappear then I think you can safely ignore them, and restore the "-k" option, if you want.
On 03/07/2018 07:30 PM, Hari Prasanth Loganathan wrote:
Thanks Juan for your response. Appreciate it. But for some reason still, I am facing the same SSL handshake failed (5). Could you please check this configuration and let me know the issue in my ovirt engine environment.
*Configuration of Apache server:*
1) httpd version,
# httpd -v Server version: Apache/2.4.6 (CentOS) Server built: Oct 19 2017 20:39:16
2) I checked the status using the following command,
# systemctl status httpd -l ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2018-03-07 23:46:32 IST; 1min 55s ago Docs: man:httpd(8) man:apachectl(8) Process: 4351 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS) Main PID: 4359 (httpd) Status: "Total requests: 264; Current requests/sec: 0.1; Current traffic: 204 B/sec" CGroup: /system.slice/httpd.service ├─4359 /usr/sbin/httpd -DFOREGROUND ├─4360 /usr/sbin/httpd -DFOREGROUND ├─4362 /usr/sbin/httpd -DFOREGROUND ├─5100 /usr/sbin/httpd -DFOREGROUND ├─5386 /usr/sbin/httpd -DFOREGROUND ├─5415 /usr/sbin/httpd -DFOREGROUND └─5416 /usr/sbin/httpd -DFOREGROUND
3) Since the httpd is pointing to the path : /usr/lib/systemd/system/httpd.service
vi /usr/lib/systemd/system/httpd.service
[Unit] Description=The Apache HTTP Server After=network.target remote-fs.target nss-lookup.target Documentation=man:httpd(8) Documentation=man:apachectl(8)
[Service] Type=notify EnvironmentFile=/etc/sysconfig/httpd ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND ExecReload=/usr/sbin/httpd $OPTIONS -k graceful ExecStop=/bin/kill -WINCH ${MAINPID} # We want systemd to give httpd some time to finish gracefully, but still want # it to kill httpd after TimeoutStopSec if something went wrong during the # graceful stop. Normally, Systemd sends SIGTERM signal right after the # ExecStop, which would kill httpd. We are sending useless SIGCONT here to give # httpd time to finish. KillSignal=SIGCONT PrivateTmp=true
[Install] WantedBy=multi-user.target
4) As per the above command I found the env file is available '/etc/sysconfig/httpd'
vi /etc/sysconfig/httpd
# # This file can be used to set additional environment variables for # the httpd process, or pass additional options to the httpd # executable. # # Note: With previous versions of httpd, the MPM could be changed by # editing an "HTTPD" variable here. With the current version, that # variable is now ignored. The MPM is a loadable module, and the # choice of MPM can be changed by editing the configuration file /etc/httpd/conf.modules.d/00-mpm.conf #
# # To pass additional options (for instance, -D definitions) to the # httpd binary at startup, set OPTIONS here. # #OPTIONS=
# # This setting ensures the httpd process is started in the "C" locale # by default. (Some modules will not behave correctly if # case-sensitive string comparisons are performed in a different # locale.) # LANG=C
5) As per the above command, I found that the conf fileis available in the path : /etc/httpd/conf.modules.d/00-mpm.conf
vi /etc/httpd/conf.modules.d/00-mpm.conf
# Select the MPM module which should be used by uncommenting exactly # one of the following LoadModule lines:
# prefork MPM: Implements a non-threaded, pre-forking web server # See: http://httpd.apache.org/docs/2.4/mod/prefork.html #LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
# worker MPM: Multi-Processing Module implementing a hybrid # multi-threaded multi-process web server # See: http://httpd.apache.org/docs/2.4/mod/worker.html # LoadModule mpm_worker_module modules/mod_mpm_worker.so
# event MPM: A variant of the worker MPM with the goal of consuming # threads only for connections with active processing # See: http://httpd.apache.org/docs/2.4/mod/event.html # #LoadModule mpm_event_module modules/mod_mpm_event.so
<IfModule mpm_worker_module> ServerLimit 1000 MaxRequestWorkers 1000 </IfModule>
As per your comment, I enabled the 'LoadModule mpm_worker_module modules/mod_mpm_worker.so' with the ServerLimit and MaxRequestWorkers as 1000 still I am facing the issue for the following command in apache benchmark test.
Completed 100 requests SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5).
NOTE : It always scales when I have concurrent request below 400
What is wrong in this apache configuration, why SSL handshake is failing for concurrent request above 400 ?
Thanks, Hari
On Wed, Mar 7, 2018 at 9:20 PM, Juan Hernández <jhernand@redhat.com> wrote:
It means that with the default configuration the Apache web server can't
serve more than 256 concurrent connections. This applies to any application that uses Apache as the web frontend, not just to oVirt. If you want to change that you have to change the MaxRequestWorkers and ServerLimit parameters, as explained here:
https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m axrequestworkers
So, go to your oVirt engine machine and create a /etc/httpd/conf.d/my.conf file with this content:
MaxRequestWorkers 1000 ServerLimit 1000
Then restart the Apache server:
# systemctl restart httpd
Then your web server should be able to handle 1000 concurrent requests, and you will probably start to find other limits, like the amount of memory and CPU that those 1000 Apache child processes will consume, the number of threads in the JBoss application server, the number of connections to the database, etc.
Let me insist a bit that if you base your benchmark solely on the number of concurrent requests or connections that the server can handle you may end up with meaningless results, as a real world application can/should use the server much better than that.
On 03/07/2018 04:33 PM, Hari Prasanth Loganathan wrote:
With the default configuration of the web server it is impossible to
handle more than 256 *connections* simultaneously. I guess that "ab" is opening a connection for each concurrent request, so when you reach request 257 the web server will just reject the connection, there is nothing that the JBoss can do about it; you have to increase the number of connections supported by the web server.
*So Does it mean that oVirt cannot serve more than 257 request? *
My question is, If its possible How to scale this and what is the configuration we need to change?
Also, we are taking a benchmark in using oVirt, So I need to find the maximum possible oVirt request. So please let me know the configuration tuning for oVirt to achieve maximum concurrent request.
Thanks, Hari
On Wed, Mar 7, 2018 at 7:25 PM, Juan Hernández <jhernand@redhat.com> wrote:
With the default configuration of the web server it is impossible to
handle more than 256 *connections* simultaneously. I guess that "ab" is opening a connection for each concurrent request, so when you reach request 257 the web server will just reject the connection, there is nothing that the JBoss can do about it; you have to increase the number of connections supported by the web server.
Or else you may want to re-consider why you want to use 1000 simultaneous connections. It may be OK for a performance test, but there are better ways to squeeze performance. For example, you could consider using HTTP pipelining, which is much more friendly for the server than so many connections. This is what we use when we need to send a large number of requests from other systems. There are examples of how to do that with the Python and Ruby SDKs here:
Python:
https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/ examples/asynchronous_inventory.py
Ruby:
https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/ sdk/examples/asynchronous_inventory.rb
On 03/07/2018 02:43 PM, Hari Prasanth Loganathan wrote:
Hi Juan,
Thanks for the response.
I agree web server can handle only limited number of concurrent requests. But Why it is failing with SSL handshake failure for few requests, Can't the JBOSS wait and serve the request? We can spare the delay but not with the request fails. So Is there a configuration in oVirt which can be tuned to achieve this?
Thanks, Hari
On Wed, Mar 7, 2018 at 7:05 PM, Juan Hernández <jhernand@redhat.com> wrote:
The first thing you will need to change for such a test is the number of
simultaneous connections accepted by the Apache web server: by default > the > max is 256. See the Apache documentation here: > > https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m > axrequestworkers > > In addition I also suggest that you consider using the "worker" > multi-processing module instead of the "prefork", as it usually works > better when talking to a Java application server, because it re-uses > connections better. > > On 03/07/2018 02:20 PM, Hari Prasanth Loganathan wrote: > > Hi Team, > > >> *Description of problem:* >> >> I am trying to achieve 1000 concurrent request to oVirt. What are >> the >> tunable parameters to achieve this? >> >> I tried to perform the benchmarking for ovirt engine using Apache >> benchmark >> using the same SSO token. >> >> ab -n 1000 -c 500 -k -H "accept: application/json" -H >> "Authorization: >> Bearer SSOTOKEN" https://172.30.56.70/ovirt-engine/ >> <https://172.30.56.70/ovirt-engine/api/vms/5440271b-afb3-48b >> b-9ff1-076fc07ebf50/statistics> >> >> When the number of concurrent request is 500, we are getting more >> than >> 100 >> failures with the following error, >> >> SSL read failed (1) - closing connection >> 139620982339352:error: >> >> NOTE: It is scaling for concurrent request below 500. >> >> I used the profiler to get the memory and CPU and it seems very >> less, >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM >> TIME+ >> COMMAND >> 30413 ovirt 20 0 4226664 882396 6776 S 126.0 23.0 27:48.53 >> java >> >> Configuration of the machine in which Ovirt is deployed : >> >> RAM - 4GB, >> Hard disk - 100GB, >> core processor - 2, >> OS - Cent7.x. >> >> In which 2GB is allocated to oVirt. >> >> >> Version-Release number of selected component (if applicable): >> >> 4.2.2 >> >> >> How reproducible: >> >> If the number of concurrent requests are above 500, we are easily >> facing >> this issue. >> >> >> *Actual results:* >> >> SSL read failed (1) - closing connection >> 139620982339352:error: >> >> *Expected results:* >> >> Request success. >> >> >> Thanks, >> Hari >> >> >> >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >> >> >> >> >
-- DISCLAIMER The information in this e-mail is confidential and may be subject to legal privilege. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you have received this communication in error, please address with the subject heading "Received in error," send to it@msystechnologies.com, then delete the e-mail and destroy any copies of it. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. The views, opinions, conclusions and other information expressed in this electronic mail and any attachments are not given or endorsed by the company unless otherwise indicated by an authorized representative independent of this message. MSys cannot guarantee that e-mail communications are secure or error-free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses, though all reasonable precautions have been taken to ensure no viruses are present in this e-mail. As our company cannot accept responsibility for any loss or damage arising from the use of this e-mail or attachments we recommend that you subject these to your virus checking procedures prior to use
But other than those SSL error messages, are the connections really failing? Can you share the results reported by "ab"? On 03/08/2018 12:16 PM, Hari Prasanth Loganathan wrote:
No Juan, It is not working with any benchmark / application tool. It fails with the same error SSL handshake failed (5).
Could you let me know the configuration of Apache web server is correct?
Thanks, Hari
On Thu, Mar 8, 2018 at 1:08 AM, Juan Hernández <jhernand@redhat.com> wrote:
If you are still having problems I am inclined to think that it is a client issue. For example, I'd try to remove the "-k" option from the "ab" command. If you use keep alive the server may decide anyhow to close the connection after certain number of requests, even if the client asks to keep it alive. Some clients don't handle that perfectly, "ab" may have that problem. If that makes the SSL error messages disappear then I think you can safely ignore them, and restore the "-k" option, if you want.
On 03/07/2018 07:30 PM, Hari Prasanth Loganathan wrote:
Thanks Juan for your response. Appreciate it. But for some reason still, I am facing the same SSL handshake failed (5). Could you please check this configuration and let me know the issue in my ovirt engine environment.
*Configuration of Apache server:*
1) httpd version,
# httpd -v Server version: Apache/2.4.6 (CentOS) Server built: Oct 19 2017 20:39:16
2) I checked the status using the following command,
# systemctl status httpd -l ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2018-03-07 23:46:32 IST; 1min 55s ago Docs: man:httpd(8) man:apachectl(8) Process: 4351 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS) Main PID: 4359 (httpd) Status: "Total requests: 264; Current requests/sec: 0.1; Current traffic: 204 B/sec" CGroup: /system.slice/httpd.service ├─4359 /usr/sbin/httpd -DFOREGROUND ├─4360 /usr/sbin/httpd -DFOREGROUND ├─4362 /usr/sbin/httpd -DFOREGROUND ├─5100 /usr/sbin/httpd -DFOREGROUND ├─5386 /usr/sbin/httpd -DFOREGROUND ├─5415 /usr/sbin/httpd -DFOREGROUND └─5416 /usr/sbin/httpd -DFOREGROUND
3) Since the httpd is pointing to the path : /usr/lib/systemd/system/httpd.service
vi /usr/lib/systemd/system/httpd.service
[Unit] Description=The Apache HTTP Server After=network.target remote-fs.target nss-lookup.target Documentation=man:httpd(8) Documentation=man:apachectl(8)
[Service] Type=notify EnvironmentFile=/etc/sysconfig/httpd ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND ExecReload=/usr/sbin/httpd $OPTIONS -k graceful ExecStop=/bin/kill -WINCH ${MAINPID} # We want systemd to give httpd some time to finish gracefully, but still want # it to kill httpd after TimeoutStopSec if something went wrong during the # graceful stop. Normally, Systemd sends SIGTERM signal right after the # ExecStop, which would kill httpd. We are sending useless SIGCONT here to give # httpd time to finish. KillSignal=SIGCONT PrivateTmp=true
[Install] WantedBy=multi-user.target
4) As per the above command I found the env file is available '/etc/sysconfig/httpd'
vi /etc/sysconfig/httpd
# # This file can be used to set additional environment variables for # the httpd process, or pass additional options to the httpd # executable. # # Note: With previous versions of httpd, the MPM could be changed by # editing an "HTTPD" variable here. With the current version, that # variable is now ignored. The MPM is a loadable module, and the # choice of MPM can be changed by editing the configuration file /etc/httpd/conf.modules.d/00-mpm.conf #
# # To pass additional options (for instance, -D definitions) to the # httpd binary at startup, set OPTIONS here. # #OPTIONS=
# # This setting ensures the httpd process is started in the "C" locale # by default. (Some modules will not behave correctly if # case-sensitive string comparisons are performed in a different # locale.) # LANG=C
5) As per the above command, I found that the conf fileis available in the path : /etc/httpd/conf.modules.d/00-mpm.conf
vi /etc/httpd/conf.modules.d/00-mpm.conf
# Select the MPM module which should be used by uncommenting exactly # one of the following LoadModule lines:
# prefork MPM: Implements a non-threaded, pre-forking web server # See: http://httpd.apache.org/docs/2.4/mod/prefork.html #LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
# worker MPM: Multi-Processing Module implementing a hybrid # multi-threaded multi-process web server # See: http://httpd.apache.org/docs/2.4/mod/worker.html # LoadModule mpm_worker_module modules/mod_mpm_worker.so
# event MPM: A variant of the worker MPM with the goal of consuming # threads only for connections with active processing # See: http://httpd.apache.org/docs/2.4/mod/event.html # #LoadModule mpm_event_module modules/mod_mpm_event.so
<IfModule mpm_worker_module> ServerLimit 1000 MaxRequestWorkers 1000 </IfModule>
As per your comment, I enabled the 'LoadModule mpm_worker_module modules/mod_mpm_worker.so' with the ServerLimit and MaxRequestWorkers as 1000 still I am facing the issue for the following command in apache benchmark test.
Completed 100 requests SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5).
NOTE : It always scales when I have concurrent request below 400
What is wrong in this apache configuration, why SSL handshake is failing for concurrent request above 400 ?
Thanks, Hari
On Wed, Mar 7, 2018 at 9:20 PM, Juan Hernández <jhernand@redhat.com> wrote:
It means that with the default configuration the Apache web server can't
serve more than 256 concurrent connections. This applies to any application that uses Apache as the web frontend, not just to oVirt. If you want to change that you have to change the MaxRequestWorkers and ServerLimit parameters, as explained here:
https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m axrequestworkers
So, go to your oVirt engine machine and create a /etc/httpd/conf.d/my.conf file with this content:
MaxRequestWorkers 1000 ServerLimit 1000
Then restart the Apache server:
# systemctl restart httpd
Then your web server should be able to handle 1000 concurrent requests, and you will probably start to find other limits, like the amount of memory and CPU that those 1000 Apache child processes will consume, the number of threads in the JBoss application server, the number of connections to the database, etc.
Let me insist a bit that if you base your benchmark solely on the number of concurrent requests or connections that the server can handle you may end up with meaningless results, as a real world application can/should use the server much better than that.
On 03/07/2018 04:33 PM, Hari Prasanth Loganathan wrote:
With the default configuration of the web server it is impossible to
handle more than 256 *connections* simultaneously. I guess that "ab" is opening a connection for each concurrent request, so when you reach request 257 the web server will just reject the connection, there is nothing that the JBoss can do about it; you have to increase the number of connections supported by the web server.
*So Does it mean that oVirt cannot serve more than 257 request? *
My question is, If its possible How to scale this and what is the configuration we need to change?
Also, we are taking a benchmark in using oVirt, So I need to find the maximum possible oVirt request. So please let me know the configuration tuning for oVirt to achieve maximum concurrent request.
Thanks, Hari
On Wed, Mar 7, 2018 at 7:25 PM, Juan Hernández <jhernand@redhat.com> wrote:
With the default configuration of the web server it is impossible to
handle more than 256 *connections* simultaneously. I guess that "ab" is opening a connection for each concurrent request, so when you reach request 257 the web server will just reject the connection, there is nothing that the JBoss can do about it; you have to increase the number of connections supported by the web server.
Or else you may want to re-consider why you want to use 1000 simultaneous connections. It may be OK for a performance test, but there are better ways to squeeze performance. For example, you could consider using HTTP pipelining, which is much more friendly for the server than so many connections. This is what we use when we need to send a large number of requests from other systems. There are examples of how to do that with the Python and Ruby SDKs here:
Python:
https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/ examples/asynchronous_inventory.py
Ruby:
https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/ sdk/examples/asynchronous_inventory.rb
On 03/07/2018 02:43 PM, Hari Prasanth Loganathan wrote:
Hi Juan,
> > Thanks for the response. > > I agree web server can handle only limited number of concurrent > requests. > But Why it is failing with SSL handshake failure for few requests, > Can't > the JBOSS wait and serve the request? We can spare the delay but not > with > the request fails. So Is there a configuration in oVirt which can be > tuned > to achieve this? > > Thanks, > Hari > > On Wed, Mar 7, 2018 at 7:05 PM, Juan Hernández <jhernand@redhat.com> > wrote: > > The first thing you will need to change for such a test is the number > of > > simultaneous connections accepted by the Apache web server: by default >> the >> max is 256. See the Apache documentation here: >> >> https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m >> axrequestworkers >> >> In addition I also suggest that you consider using the "worker" >> multi-processing module instead of the "prefork", as it usually works >> better when talking to a Java application server, because it re-uses >> connections better. >> >> On 03/07/2018 02:20 PM, Hari Prasanth Loganathan wrote: >> >> Hi Team, >> >> >>> *Description of problem:* >>> >>> I am trying to achieve 1000 concurrent request to oVirt. What are >>> the >>> tunable parameters to achieve this? >>> >>> I tried to perform the benchmarking for ovirt engine using Apache >>> benchmark >>> using the same SSO token. >>> >>> ab -n 1000 -c 500 -k -H "accept: application/json" -H >>> "Authorization: >>> Bearer SSOTOKEN" https://172.30.56.70/ovirt-engine/ >>> <https://172.30.56.70/ovirt-engine/api/vms/5440271b-afb3-48b >>> b-9ff1-076fc07ebf50/statistics> >>> >>> When the number of concurrent request is 500, we are getting more >>> than >>> 100 >>> failures with the following error, >>> >>> SSL read failed (1) - closing connection >>> 139620982339352:error: >>> >>> NOTE: It is scaling for concurrent request below 500. >>> >>> I used the profiler to get the memory and CPU and it seems very >>> less, >>> >>> PID USER PR NI VIRT RES SHR S %CPU %MEM >>> TIME+ >>> COMMAND >>> 30413 ovirt 20 0 4226664 882396 6776 S 126.0 23.0 27:48.53 >>> java >>> >>> Configuration of the machine in which Ovirt is deployed : >>> >>> RAM - 4GB, >>> Hard disk - 100GB, >>> core processor - 2, >>> OS - Cent7.x. >>> >>> In which 2GB is allocated to oVirt. >>> >>> >>> Version-Release number of selected component (if applicable): >>> >>> 4.2.2 >>> >>> >>> How reproducible: >>> >>> If the number of concurrent requests are above 500, we are easily >>> facing >>> this issue. >>> >>> >>> *Actual results:* >>> >>> SSL read failed (1) - closing connection >>> 139620982339352:error: >>> >>> *Expected results:* >>> >>> Request success. >>> >>> >>> Thanks, >>> Hari >>> >>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >>> >>> >>> >> >
This is the only error message we received from ab. I googled it and found that it is due to the connection drop. So It would be Great, If you could check my Apache server configuration I shared in the thread and let me know your thoughts on this. Thanks, Hari On Thu, Mar 8, 2018 at 4:56 PM, Juan Hernández <jhernand@redhat.com> wrote:
But other than those SSL error messages, are the connections really failing? Can you share the results reported by "ab"?
On 03/08/2018 12:16 PM, Hari Prasanth Loganathan wrote:
No Juan, It is not working with any benchmark / application tool. It fails with the same error SSL handshake failed (5).
Could you let me know the configuration of Apache web server is correct?
Thanks, Hari
On Thu, Mar 8, 2018 at 1:08 AM, Juan Hernández <jhernand@redhat.com> wrote:
If you are still having problems I am inclined to think that it is a
client issue. For example, I'd try to remove the "-k" option from the "ab" command. If you use keep alive the server may decide anyhow to close the connection after certain number of requests, even if the client asks to keep it alive. Some clients don't handle that perfectly, "ab" may have that problem. If that makes the SSL error messages disappear then I think you can safely ignore them, and restore the "-k" option, if you want.
On 03/07/2018 07:30 PM, Hari Prasanth Loganathan wrote:
Thanks Juan for your response. Appreciate it.
But for some reason still, I am facing the same SSL handshake failed (5). Could you please check this configuration and let me know the issue in my ovirt engine environment.
*Configuration of Apache server:*
1) httpd version,
# httpd -v Server version: Apache/2.4.6 (CentOS) Server built: Oct 19 2017 20:39:16
2) I checked the status using the following command,
# systemctl status httpd -l ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2018-03-07 23:46:32 IST; 1min 55s ago Docs: man:httpd(8) man:apachectl(8) Process: 4351 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS) Main PID: 4359 (httpd) Status: "Total requests: 264; Current requests/sec: 0.1; Current traffic: 204 B/sec" CGroup: /system.slice/httpd.service ├─4359 /usr/sbin/httpd -DFOREGROUND ├─4360 /usr/sbin/httpd -DFOREGROUND ├─4362 /usr/sbin/httpd -DFOREGROUND ├─5100 /usr/sbin/httpd -DFOREGROUND ├─5386 /usr/sbin/httpd -DFOREGROUND ├─5415 /usr/sbin/httpd -DFOREGROUND └─5416 /usr/sbin/httpd -DFOREGROUND
3) Since the httpd is pointing to the path : /usr/lib/systemd/system/httpd.service
vi /usr/lib/systemd/system/httpd.service
[Unit] Description=The Apache HTTP Server After=network.target remote-fs.target nss-lookup.target Documentation=man:httpd(8) Documentation=man:apachectl(8)
[Service] Type=notify EnvironmentFile=/etc/sysconfig/httpd ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND ExecReload=/usr/sbin/httpd $OPTIONS -k graceful ExecStop=/bin/kill -WINCH ${MAINPID} # We want systemd to give httpd some time to finish gracefully, but still want # it to kill httpd after TimeoutStopSec if something went wrong during the # graceful stop. Normally, Systemd sends SIGTERM signal right after the # ExecStop, which would kill httpd. We are sending useless SIGCONT here to give # httpd time to finish. KillSignal=SIGCONT PrivateTmp=true
[Install] WantedBy=multi-user.target
4) As per the above command I found the env file is available '/etc/sysconfig/httpd'
vi /etc/sysconfig/httpd
# # This file can be used to set additional environment variables for # the httpd process, or pass additional options to the httpd # executable. # # Note: With previous versions of httpd, the MPM could be changed by # editing an "HTTPD" variable here. With the current version, that # variable is now ignored. The MPM is a loadable module, and the # choice of MPM can be changed by editing the configuration file /etc/httpd/conf.modules.d/00-mpm.conf #
# # To pass additional options (for instance, -D definitions) to the # httpd binary at startup, set OPTIONS here. # #OPTIONS=
# # This setting ensures the httpd process is started in the "C" locale # by default. (Some modules will not behave correctly if # case-sensitive string comparisons are performed in a different # locale.) # LANG=C
5) As per the above command, I found that the conf fileis available in the path : /etc/httpd/conf.modules.d/00-mpm.conf
vi /etc/httpd/conf.modules.d/00-mpm.conf
# Select the MPM module which should be used by uncommenting exactly # one of the following LoadModule lines:
# prefork MPM: Implements a non-threaded, pre-forking web server # See: http://httpd.apache.org/docs/2.4/mod/prefork.html #LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
# worker MPM: Multi-Processing Module implementing a hybrid # multi-threaded multi-process web server # See: http://httpd.apache.org/docs/2.4/mod/worker.html # LoadModule mpm_worker_module modules/mod_mpm_worker.so
# event MPM: A variant of the worker MPM with the goal of consuming # threads only for connections with active processing # See: http://httpd.apache.org/docs/2.4/mod/event.html # #LoadModule mpm_event_module modules/mod_mpm_event.so
<IfModule mpm_worker_module> ServerLimit 1000 MaxRequestWorkers 1000 </IfModule>
As per your comment, I enabled the 'LoadModule mpm_worker_module modules/mod_mpm_worker.so' with the ServerLimit and MaxRequestWorkers as 1000 still I am facing the issue for the following command in apache benchmark test.
Completed 100 requests SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5).
NOTE : It always scales when I have concurrent request below 400
What is wrong in this apache configuration, why SSL handshake is failing for concurrent request above 400 ?
Thanks, Hari
On Wed, Mar 7, 2018 at 9:20 PM, Juan Hernández <jhernand@redhat.com> wrote:
It means that with the default configuration the Apache web server can't
serve more than 256 concurrent connections. This applies to any application that uses Apache as the web frontend, not just to oVirt. If you want to change that you have to change the MaxRequestWorkers and ServerLimit parameters, as explained here:
https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m axrequestworkers
So, go to your oVirt engine machine and create a /etc/httpd/conf.d/my.conf file with this content:
MaxRequestWorkers 1000 ServerLimit 1000
Then restart the Apache server:
# systemctl restart httpd
Then your web server should be able to handle 1000 concurrent requests, and you will probably start to find other limits, like the amount of memory and CPU that those 1000 Apache child processes will consume, the number of threads in the JBoss application server, the number of connections to the database, etc.
Let me insist a bit that if you base your benchmark solely on the number of concurrent requests or connections that the server can handle you may end up with meaningless results, as a real world application can/should use the server much better than that.
On 03/07/2018 04:33 PM, Hari Prasanth Loganathan wrote:
With the default configuration of the web server it is impossible to
handle more than 256 *connections* simultaneously. I guess that "ab" is opening a connection for each concurrent request, so when you reach request 257 the web server will just reject the connection, there is nothing that the JBoss can do about it; you have to increase the number of connections supported by the web server.
*So Does it mean that oVirt cannot serve more than 257 request? *
My question is, If its possible How to scale this and what is the configuration we need to change?
Also, we are taking a benchmark in using oVirt, So I need to find the maximum possible oVirt request. So please let me know the configuration tuning for oVirt to achieve maximum concurrent request.
Thanks, Hari
On Wed, Mar 7, 2018 at 7:25 PM, Juan Hernández <jhernand@redhat.com> wrote:
With the default configuration of the web server it is impossible to
handle more than 256 *connections* simultaneously. I guess that "ab" > is > opening a connection for each concurrent request, so when you reach > request > 257 the web server will just reject the connection, there is nothing > that > the JBoss can do about it; you have to increase the number of > connections > supported by the web server. > > Or else you may want to re-consider why you want to use 1000 > simultaneous > connections. It may be OK for a performance test, but there are > better > ways > to squeeze performance. For example, you could consider using HTTP > pipelining, which is much more friendly for the server than so many > connections. This is what we use when we need to send a large number > of > requests from other systems. There are examples of how to do that > with > the > Python and Ruby SDKs here: > > Python: > > https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/ > examples/asynchronous_inventory.py > > Ruby: > > https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/ > sdk/examples/asynchronous_inventory.rb > > > On 03/07/2018 02:43 PM, Hari Prasanth Loganathan wrote: > > Hi Juan, > > >> Thanks for the response. >> >> I agree web server can handle only limited number of concurrent >> requests. >> But Why it is failing with SSL handshake failure for few requests, >> Can't >> the JBOSS wait and serve the request? We can spare the delay but not >> with >> the request fails. So Is there a configuration in oVirt which can be >> tuned >> to achieve this? >> >> Thanks, >> Hari >> >> On Wed, Mar 7, 2018 at 7:05 PM, Juan Hernández <jhernand@redhat.com >> > >> wrote: >> >> The first thing you will need to change for such a test is the >> number >> of >> >> simultaneous connections accepted by the Apache web server: by >> default >> >>> the >>> max is 256. See the Apache documentation here: >>> >>> https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m >>> axrequestworkers >>> >>> In addition I also suggest that you consider using the "worker" >>> multi-processing module instead of the "prefork", as it usually >>> works >>> better when talking to a Java application server, because it >>> re-uses >>> connections better. >>> >>> On 03/07/2018 02:20 PM, Hari Prasanth Loganathan wrote: >>> >>> Hi Team, >>> >>> >>> *Description of problem:* >>>> >>>> I am trying to achieve 1000 concurrent request to oVirt. What are >>>> the >>>> tunable parameters to achieve this? >>>> >>>> I tried to perform the benchmarking for ovirt engine using Apache >>>> benchmark >>>> using the same SSO token. >>>> >>>> ab -n 1000 -c 500 -k -H "accept: application/json" -H >>>> "Authorization: >>>> Bearer SSOTOKEN" https://172.30.56.70/ovirt-engine/ >>>> <https://172.30.56.70/ovirt-engine/api/vms/5440271b-afb3-48b >>>> b-9ff1-076fc07ebf50/statistics> >>>> >>>> When the number of concurrent request is 500, we are getting more >>>> than >>>> 100 >>>> failures with the following error, >>>> >>>> SSL read failed (1) - closing connection >>>> 139620982339352:error: >>>> >>>> NOTE: It is scaling for concurrent request below 500. >>>> >>>> I used the profiler to get the memory and CPU and it seems very >>>> less, >>>> >>>> PID USER PR NI VIRT RES SHR S %CPU %MEM >>>> TIME+ >>>> COMMAND >>>> 30413 ovirt 20 0 4226664 882396 6776 S 126.0 23.0 >>>> 27:48.53 >>>> java >>>> >>>> Configuration of the machine in which Ovirt is deployed : >>>> >>>> RAM - 4GB, >>>> Hard disk - 100GB, >>>> core processor - 2, >>>> OS - Cent7.x. >>>> >>>> In which 2GB is allocated to oVirt. >>>> >>>> >>>> Version-Release number of selected component (if applicable): >>>> >>>> 4.2.2 >>>> >>>> >>>> How reproducible: >>>> >>>> If the number of concurrent requests are above 500, we are easily >>>> facing >>>> this issue. >>>> >>>> >>>> *Actual results:* >>>> >>>> SSL read failed (1) - closing connection >>>> 139620982339352:error: >>>> >>>> *Expected results:* >>>> >>>> Request success. >>>> >>>> >>>> Thanks, >>>> Hari >>>> >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users@ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >>>> >>>> >>>> >>>> >>>> >>> >> >
-- DISCLAIMER The information in this e-mail is confidential and may be subject to legal privilege. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you have received this communication in error, please address with the subject heading "Received in error," send to it@msystechnologies.com, then delete the e-mail and destroy any copies of it. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. The views, opinions, conclusions and other information expressed in this electronic mail and any attachments are not given or endorsed by the company unless otherwise indicated by an authorized representative independent of this message. MSys cannot guarantee that e-mail communications are secure or error-free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses, though all reasonable precautions have been taken to ensure no viruses are present in this e-mail. As our company cannot accept responsibility for any loss or damage arising from the use of this e-mail or attachments we recommend that you subject these to your virus checking procedures prior to use
I think the configuration is good. There may be some connections that the server is closing before the client expected, but that is normal, in my opinion. On 03/08/2018 12:34 PM, Hari Prasanth Loganathan wrote:
This is the only error message we received from ab.
I googled it and found that it is due to the connection drop. So It would be Great, If you could check my Apache server configuration I shared in the thread and let me know your thoughts on this.
Thanks, Hari
On Thu, Mar 8, 2018 at 4:56 PM, Juan Hernández <jhernand@redhat.com> wrote:
But other than those SSL error messages, are the connections really failing? Can you share the results reported by "ab"?
On 03/08/2018 12:16 PM, Hari Prasanth Loganathan wrote:
No Juan, It is not working with any benchmark / application tool. It fails with the same error SSL handshake failed (5).
Could you let me know the configuration of Apache web server is correct?
Thanks, Hari
On Thu, Mar 8, 2018 at 1:08 AM, Juan Hernández <jhernand@redhat.com> wrote:
If you are still having problems I am inclined to think that it is a
client issue. For example, I'd try to remove the "-k" option from the "ab" command. If you use keep alive the server may decide anyhow to close the connection after certain number of requests, even if the client asks to keep it alive. Some clients don't handle that perfectly, "ab" may have that problem. If that makes the SSL error messages disappear then I think you can safely ignore them, and restore the "-k" option, if you want.
On 03/07/2018 07:30 PM, Hari Prasanth Loganathan wrote:
Thanks Juan for your response. Appreciate it.
But for some reason still, I am facing the same SSL handshake failed (5). Could you please check this configuration and let me know the issue in my ovirt engine environment.
*Configuration of Apache server:*
1) httpd version,
# httpd -v Server version: Apache/2.4.6 (CentOS) Server built: Oct 19 2017 20:39:16
2) I checked the status using the following command,
# systemctl status httpd -l ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2018-03-07 23:46:32 IST; 1min 55s ago Docs: man:httpd(8) man:apachectl(8) Process: 4351 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS) Main PID: 4359 (httpd) Status: "Total requests: 264; Current requests/sec: 0.1; Current traffic: 204 B/sec" CGroup: /system.slice/httpd.service ├─4359 /usr/sbin/httpd -DFOREGROUND ├─4360 /usr/sbin/httpd -DFOREGROUND ├─4362 /usr/sbin/httpd -DFOREGROUND ├─5100 /usr/sbin/httpd -DFOREGROUND ├─5386 /usr/sbin/httpd -DFOREGROUND ├─5415 /usr/sbin/httpd -DFOREGROUND └─5416 /usr/sbin/httpd -DFOREGROUND
3) Since the httpd is pointing to the path : /usr/lib/systemd/system/httpd.service
vi /usr/lib/systemd/system/httpd.service
[Unit] Description=The Apache HTTP Server After=network.target remote-fs.target nss-lookup.target Documentation=man:httpd(8) Documentation=man:apachectl(8)
[Service] Type=notify EnvironmentFile=/etc/sysconfig/httpd ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND ExecReload=/usr/sbin/httpd $OPTIONS -k graceful ExecStop=/bin/kill -WINCH ${MAINPID} # We want systemd to give httpd some time to finish gracefully, but still want # it to kill httpd after TimeoutStopSec if something went wrong during the # graceful stop. Normally, Systemd sends SIGTERM signal right after the # ExecStop, which would kill httpd. We are sending useless SIGCONT here to give # httpd time to finish. KillSignal=SIGCONT PrivateTmp=true
[Install] WantedBy=multi-user.target
4) As per the above command I found the env file is available '/etc/sysconfig/httpd'
vi /etc/sysconfig/httpd
# # This file can be used to set additional environment variables for # the httpd process, or pass additional options to the httpd # executable. # # Note: With previous versions of httpd, the MPM could be changed by # editing an "HTTPD" variable here. With the current version, that # variable is now ignored. The MPM is a loadable module, and the # choice of MPM can be changed by editing the configuration file /etc/httpd/conf.modules.d/00-mpm.conf #
# # To pass additional options (for instance, -D definitions) to the # httpd binary at startup, set OPTIONS here. # #OPTIONS=
# # This setting ensures the httpd process is started in the "C" locale # by default. (Some modules will not behave correctly if # case-sensitive string comparisons are performed in a different # locale.) # LANG=C
5) As per the above command, I found that the conf fileis available in the path : /etc/httpd/conf.modules.d/00-mpm.conf
vi /etc/httpd/conf.modules.d/00-mpm.conf
# Select the MPM module which should be used by uncommenting exactly # one of the following LoadModule lines:
# prefork MPM: Implements a non-threaded, pre-forking web server # See: http://httpd.apache.org/docs/2.4/mod/prefork.html #LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
# worker MPM: Multi-Processing Module implementing a hybrid # multi-threaded multi-process web server # See: http://httpd.apache.org/docs/2.4/mod/worker.html # LoadModule mpm_worker_module modules/mod_mpm_worker.so
# event MPM: A variant of the worker MPM with the goal of consuming # threads only for connections with active processing # See: http://httpd.apache.org/docs/2.4/mod/event.html # #LoadModule mpm_event_module modules/mod_mpm_event.so
<IfModule mpm_worker_module> ServerLimit 1000 MaxRequestWorkers 1000 </IfModule>
As per your comment, I enabled the 'LoadModule mpm_worker_module modules/mod_mpm_worker.so' with the ServerLimit and MaxRequestWorkers as 1000 still I am facing the issue for the following command in apache benchmark test.
Completed 100 requests SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5).
NOTE : It always scales when I have concurrent request below 400
What is wrong in this apache configuration, why SSL handshake is failing for concurrent request above 400 ?
Thanks, Hari
On Wed, Mar 7, 2018 at 9:20 PM, Juan Hernández <jhernand@redhat.com> wrote:
It means that with the default configuration the Apache web server can't
serve more than 256 concurrent connections. This applies to any application that uses Apache as the web frontend, not just to oVirt. If you want to change that you have to change the MaxRequestWorkers and ServerLimit parameters, as explained here:
https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m axrequestworkers
So, go to your oVirt engine machine and create a /etc/httpd/conf.d/my.conf file with this content:
MaxRequestWorkers 1000 ServerLimit 1000
Then restart the Apache server:
# systemctl restart httpd
Then your web server should be able to handle 1000 concurrent requests, and you will probably start to find other limits, like the amount of memory and CPU that those 1000 Apache child processes will consume, the number of threads in the JBoss application server, the number of connections to the database, etc.
Let me insist a bit that if you base your benchmark solely on the number of concurrent requests or connections that the server can handle you may end up with meaningless results, as a real world application can/should use the server much better than that.
On 03/07/2018 04:33 PM, Hari Prasanth Loganathan wrote:
With the default configuration of the web server it is impossible to
> handle > more than 256 *connections* simultaneously. I guess that "ab" is > opening a > connection for each concurrent request, so when you reach request 257 > the > web server will just reject the connection, there is nothing that the > JBoss > can do about it; you have to increase the number of connections > supported > by the web server. > > *So Does it mean that oVirt cannot serve more than 257 request? * > > My question is, If its possible How to scale this and what is the > configuration we need to change? > > Also, we are taking a benchmark in using oVirt, So I need to find the > maximum possible oVirt request. So please let me know the > configuration > tuning for oVirt to achieve maximum concurrent request. > > Thanks, > Hari > > On Wed, Mar 7, 2018 at 7:25 PM, Juan Hernández <jhernand@redhat.com> > wrote: > > With the default configuration of the web server it is impossible to > > handle more than 256 *connections* simultaneously. I guess that "ab" >> is >> opening a connection for each concurrent request, so when you reach >> request >> 257 the web server will just reject the connection, there is nothing >> that >> the JBoss can do about it; you have to increase the number of >> connections >> supported by the web server. >> >> Or else you may want to re-consider why you want to use 1000 >> simultaneous >> connections. It may be OK for a performance test, but there are >> better >> ways >> to squeeze performance. For example, you could consider using HTTP >> pipelining, which is much more friendly for the server than so many >> connections. This is what we use when we need to send a large number >> of >> requests from other systems. There are examples of how to do that >> with >> the >> Python and Ruby SDKs here: >> >> Python: >> >> https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/ >> examples/asynchronous_inventory.py >> >> Ruby: >> >> https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/ >> sdk/examples/asynchronous_inventory.rb >> >> >> On 03/07/2018 02:43 PM, Hari Prasanth Loganathan wrote: >> >> Hi Juan, >> >> >>> Thanks for the response. >>> >>> I agree web server can handle only limited number of concurrent >>> requests. >>> But Why it is failing with SSL handshake failure for few requests, >>> Can't >>> the JBOSS wait and serve the request? We can spare the delay but not >>> with >>> the request fails. So Is there a configuration in oVirt which can be >>> tuned >>> to achieve this? >>> >>> Thanks, >>> Hari >>> >>> On Wed, Mar 7, 2018 at 7:05 PM, Juan Hernández <jhernand@redhat.com >>>> >>> wrote: >>> >>> The first thing you will need to change for such a test is the >>> number >>> of >>> >>> simultaneous connections accepted by the Apache web server: by >>> default >>> >>>> the >>>> max is 256. See the Apache documentation here: >>>> >>>> https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m >>>> axrequestworkers >>>> >>>> In addition I also suggest that you consider using the "worker" >>>> multi-processing module instead of the "prefork", as it usually >>>> works >>>> better when talking to a Java application server, because it >>>> re-uses >>>> connections better. >>>> >>>> On 03/07/2018 02:20 PM, Hari Prasanth Loganathan wrote: >>>> >>>> Hi Team, >>>> >>>> >>>> *Description of problem:* >>>>> >>>>> I am trying to achieve 1000 concurrent request to oVirt. What are >>>>> the >>>>> tunable parameters to achieve this? >>>>> >>>>> I tried to perform the benchmarking for ovirt engine using Apache >>>>> benchmark >>>>> using the same SSO token. >>>>> >>>>> ab -n 1000 -c 500 -k -H "accept: application/json" -H >>>>> "Authorization: >>>>> Bearer SSOTOKEN" https://172.30.56.70/ovirt-engine/ >>>>> <https://172.30.56.70/ovirt-engine/api/vms/5440271b-afb3-48b >>>>> b-9ff1-076fc07ebf50/statistics> >>>>> >>>>> When the number of concurrent request is 500, we are getting more >>>>> than >>>>> 100 >>>>> failures with the following error, >>>>> >>>>> SSL read failed (1) - closing connection >>>>> 139620982339352:error: >>>>> >>>>> NOTE: It is scaling for concurrent request below 500. >>>>> >>>>> I used the profiler to get the memory and CPU and it seems very >>>>> less, >>>>> >>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM >>>>> TIME+ >>>>> COMMAND >>>>> 30413 ovirt 20 0 4226664 882396 6776 S 126.0 23.0 >>>>> 27:48.53 >>>>> java >>>>> >>>>> Configuration of the machine in which Ovirt is deployed : >>>>> >>>>> RAM - 4GB, >>>>> Hard disk - 100GB, >>>>> core processor - 2, >>>>> OS - Cent7.x. >>>>> >>>>> In which 2GB is allocated to oVirt. >>>>> >>>>> >>>>> Version-Release number of selected component (if applicable): >>>>> >>>>> 4.2.2 >>>>> >>>>> >>>>> How reproducible: >>>>> >>>>> If the number of concurrent requests are above 500, we are easily >>>>> facing >>>>> this issue. >>>>> >>>>> >>>>> *Actual results:* >>>>> >>>>> SSL read failed (1) - closing connection >>>>> 139620982339352:error: >>>>> >>>>> *Expected results:* >>>>> >>>>> Request success. >>>>> >>>>> >>>>> Thanks, >>>>> Hari >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users@ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>> >> >
Hi Juan, Now we got a clue, I try to hit 100 concurrent request in which 1 request got failed with the 503 error, I observed this in apache ssl_access.log 172.30.36.167 - - [08/Mar/2018:17:22:42 +0530] "GET /ovirt-engine/api/vms/ HTTP/1.1" 503 299 Is it thrown from Apache / ovirt engine? On Thu, Mar 8, 2018 at 5:15 PM, Juan Hernández <jhernand@redhat.com> wrote:
I think the configuration is good. There may be some connections that the server is closing before the client expected, but that is normal, in my opinion.
On 03/08/2018 12:34 PM, Hari Prasanth Loganathan wrote:
This is the only error message we received from ab.
I googled it and found that it is due to the connection drop. So It would be Great, If you could check my Apache server configuration I shared in the thread and let me know your thoughts on this.
Thanks, Hari
On Thu, Mar 8, 2018 at 4:56 PM, Juan Hernández <jhernand@redhat.com> wrote:
But other than those SSL error messages, are the connections really
failing? Can you share the results reported by "ab"?
On 03/08/2018 12:16 PM, Hari Prasanth Loganathan wrote:
No Juan, It is not working with any benchmark / application tool. It
fails with the same error SSL handshake failed (5).
Could you let me know the configuration of Apache web server is correct?
Thanks, Hari
On Thu, Mar 8, 2018 at 1:08 AM, Juan Hernández <jhernand@redhat.com> wrote:
If you are still having problems I am inclined to think that it is a
client issue. For example, I'd try to remove the "-k" option from the "ab" command. If you use keep alive the server may decide anyhow to close the connection after certain number of requests, even if the client asks to keep it alive. Some clients don't handle that perfectly, "ab" may have that problem. If that makes the SSL error messages disappear then I think you can safely ignore them, and restore the "-k" option, if you want.
On 03/07/2018 07:30 PM, Hari Prasanth Loganathan wrote:
Thanks Juan for your response. Appreciate it.
But for some reason still, I am facing the same SSL handshake failed (5). Could you please check this configuration and let me know the issue in my ovirt engine environment.
*Configuration of Apache server:*
1) httpd version,
# httpd -v Server version: Apache/2.4.6 (CentOS) Server built: Oct 19 2017 20:39:16
2) I checked the status using the following command,
# systemctl status httpd -l ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2018-03-07 23:46:32 IST; 1min 55s ago Docs: man:httpd(8) man:apachectl(8) Process: 4351 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS) Main PID: 4359 (httpd) Status: "Total requests: 264; Current requests/sec: 0.1; Current traffic: 204 B/sec" CGroup: /system.slice/httpd.service ├─4359 /usr/sbin/httpd -DFOREGROUND ├─4360 /usr/sbin/httpd -DFOREGROUND ├─4362 /usr/sbin/httpd -DFOREGROUND ├─5100 /usr/sbin/httpd -DFOREGROUND ├─5386 /usr/sbin/httpd -DFOREGROUND ├─5415 /usr/sbin/httpd -DFOREGROUND └─5416 /usr/sbin/httpd -DFOREGROUND
3) Since the httpd is pointing to the path : /usr/lib/systemd/system/httpd.service
vi /usr/lib/systemd/system/httpd.service
[Unit] Description=The Apache HTTP Server After=network.target remote-fs.target nss-lookup.target Documentation=man:httpd(8) Documentation=man:apachectl(8)
[Service] Type=notify EnvironmentFile=/etc/sysconfig/httpd ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND ExecReload=/usr/sbin/httpd $OPTIONS -k graceful ExecStop=/bin/kill -WINCH ${MAINPID} # We want systemd to give httpd some time to finish gracefully, but still want # it to kill httpd after TimeoutStopSec if something went wrong during the # graceful stop. Normally, Systemd sends SIGTERM signal right after the # ExecStop, which would kill httpd. We are sending useless SIGCONT here to give # httpd time to finish. KillSignal=SIGCONT PrivateTmp=true
[Install] WantedBy=multi-user.target
4) As per the above command I found the env file is available '/etc/sysconfig/httpd'
vi /etc/sysconfig/httpd
# # This file can be used to set additional environment variables for # the httpd process, or pass additional options to the httpd # executable. # # Note: With previous versions of httpd, the MPM could be changed by # editing an "HTTPD" variable here. With the current version, that # variable is now ignored. The MPM is a loadable module, and the # choice of MPM can be changed by editing the configuration file /etc/httpd/conf.modules.d/00-mpm.conf #
# # To pass additional options (for instance, -D definitions) to the # httpd binary at startup, set OPTIONS here. # #OPTIONS=
# # This setting ensures the httpd process is started in the "C" locale # by default. (Some modules will not behave correctly if # case-sensitive string comparisons are performed in a different # locale.) # LANG=C
5) As per the above command, I found that the conf fileis available in the path : /etc/httpd/conf.modules.d/00-mpm.conf
vi /etc/httpd/conf.modules.d/00-mpm.conf
# Select the MPM module which should be used by uncommenting exactly # one of the following LoadModule lines:
# prefork MPM: Implements a non-threaded, pre-forking web server # See: http://httpd.apache.org/docs/2.4/mod/prefork.html #LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
# worker MPM: Multi-Processing Module implementing a hybrid # multi-threaded multi-process web server # See: http://httpd.apache.org/docs/2.4/mod/worker.html # LoadModule mpm_worker_module modules/mod_mpm_worker.so
# event MPM: A variant of the worker MPM with the goal of consuming # threads only for connections with active processing # See: http://httpd.apache.org/docs/2.4/mod/event.html # #LoadModule mpm_event_module modules/mod_mpm_event.so
<IfModule mpm_worker_module> ServerLimit 1000 MaxRequestWorkers 1000 </IfModule>
As per your comment, I enabled the 'LoadModule mpm_worker_module modules/mod_mpm_worker.so' with the ServerLimit and MaxRequestWorkers as 1000 still I am facing the issue for the following command in apache benchmark test.
Completed 100 requests SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5).
NOTE : It always scales when I have concurrent request below 400
What is wrong in this apache configuration, why SSL handshake is failing for concurrent request above 400 ?
Thanks, Hari
On Wed, Mar 7, 2018 at 9:20 PM, Juan Hernández <jhernand@redhat.com> wrote:
It means that with the default configuration the Apache web server can't
serve more than 256 concurrent connections. This applies to any > application > that uses Apache as the web frontend, not just to oVirt. If you want > to > change that you have to change the MaxRequestWorkers and ServerLimit > parameters, as explained here: > > https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m > axrequestworkers > > So, go to your oVirt engine machine and create a > /etc/httpd/conf.d/my.conf > file with this content: > > MaxRequestWorkers 1000 > ServerLimit 1000 > > Then restart the Apache server: > > # systemctl restart httpd > > Then your web server should be able to handle 1000 concurrent > requests, > and you will probably start to find other limits, like the amount of > memory > and CPU that those 1000 Apache child processes will consume, the > number > of > threads in the JBoss application server, the number of connections to > the > database, etc. > > Let me insist a bit that if you base your benchmark solely on the > number > of concurrent requests or connections that the server can handle you > may > end up with meaningless results, as a real world application > can/should > use > the server much better than that. > > On 03/07/2018 04:33 PM, Hari Prasanth Loganathan wrote: > > With the default configuration of the web server it is impossible to > > handle >> more than 256 *connections* simultaneously. I guess that "ab" is >> opening a >> connection for each concurrent request, so when you reach request >> 257 >> the >> web server will just reject the connection, there is nothing that >> the >> JBoss >> can do about it; you have to increase the number of connections >> supported >> by the web server. >> >> *So Does it mean that oVirt cannot serve more than 257 request? * >> >> My question is, If its possible How to scale this and what is the >> configuration we need to change? >> >> Also, we are taking a benchmark in using oVirt, So I need to find >> the >> maximum possible oVirt request. So please let me know the >> configuration >> tuning for oVirt to achieve maximum concurrent request. >> >> Thanks, >> Hari >> >> On Wed, Mar 7, 2018 at 7:25 PM, Juan Hernández <jhernand@redhat.com >> > >> wrote: >> >> With the default configuration of the web server it is impossible to >> >> handle more than 256 *connections* simultaneously. I guess that "ab" >> >>> is >>> opening a connection for each concurrent request, so when you reach >>> request >>> 257 the web server will just reject the connection, there is >>> nothing >>> that >>> the JBoss can do about it; you have to increase the number of >>> connections >>> supported by the web server. >>> >>> Or else you may want to re-consider why you want to use 1000 >>> simultaneous >>> connections. It may be OK for a performance test, but there are >>> better >>> ways >>> to squeeze performance. For example, you could consider using HTTP >>> pipelining, which is much more friendly for the server than so many >>> connections. This is what we use when we need to send a large >>> number >>> of >>> requests from other systems. There are examples of how to do that >>> with >>> the >>> Python and Ruby SDKs here: >>> >>> Python: >>> >>> https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/ >>> examples/asynchronous_inventory.py >>> >>> Ruby: >>> >>> https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/ >>> sdk/examples/asynchronous_inventory.rb >>> >>> >>> On 03/07/2018 02:43 PM, Hari Prasanth Loganathan wrote: >>> >>> Hi Juan, >>> >>> >>> Thanks for the response. >>>> >>>> I agree web server can handle only limited number of concurrent >>>> requests. >>>> But Why it is failing with SSL handshake failure for few requests, >>>> Can't >>>> the JBOSS wait and serve the request? We can spare the delay but >>>> not >>>> with >>>> the request fails. So Is there a configuration in oVirt which can >>>> be >>>> tuned >>>> to achieve this? >>>> >>>> Thanks, >>>> Hari >>>> >>>> On Wed, Mar 7, 2018 at 7:05 PM, Juan Hernández < >>>> jhernand@redhat.com >>>> >>>>> >>>>> wrote: >>>> >>>> The first thing you will need to change for such a test is the >>>> number >>>> of >>>> >>>> simultaneous connections accepted by the Apache web server: by >>>> default >>>> >>>> the >>>>> max is 256. See the Apache documentation here: >>>>> >>>>> https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m >>>>> axrequestworkers >>>>> >>>>> In addition I also suggest that you consider using the "worker" >>>>> multi-processing module instead of the "prefork", as it usually >>>>> works >>>>> better when talking to a Java application server, because it >>>>> re-uses >>>>> connections better. >>>>> >>>>> On 03/07/2018 02:20 PM, Hari Prasanth Loganathan wrote: >>>>> >>>>> Hi Team, >>>>> >>>>> >>>>> *Description of problem:* >>>>> >>>>>> >>>>>> I am trying to achieve 1000 concurrent request to oVirt. What >>>>>> are >>>>>> the >>>>>> tunable parameters to achieve this? >>>>>> >>>>>> I tried to perform the benchmarking for ovirt engine using >>>>>> Apache >>>>>> benchmark >>>>>> using the same SSO token. >>>>>> >>>>>> ab -n 1000 -c 500 -k -H "accept: application/json" -H >>>>>> "Authorization: >>>>>> Bearer SSOTOKEN" https://172.30.56.70/ovirt-engine/ >>>>>> <https://172.30.56.70/ovirt-engine/api/vms/5440271b-afb3-48b >>>>>> b-9ff1-076fc07ebf50/statistics> >>>>>> >>>>>> When the number of concurrent request is 500, we are getting >>>>>> more >>>>>> than >>>>>> 100 >>>>>> failures with the following error, >>>>>> >>>>>> SSL read failed (1) - closing connection >>>>>> 139620982339352:error: >>>>>> >>>>>> NOTE: It is scaling for concurrent request below 500. >>>>>> >>>>>> I used the profiler to get the memory and CPU and it seems very >>>>>> less, >>>>>> >>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM >>>>>> TIME+ >>>>>> COMMAND >>>>>> 30413 ovirt 20 0 4226664 882396 6776 S 126.0 23.0 >>>>>> 27:48.53 >>>>>> java >>>>>> >>>>>> Configuration of the machine in which Ovirt is deployed : >>>>>> >>>>>> RAM - 4GB, >>>>>> Hard disk - 100GB, >>>>>> core processor - 2, >>>>>> OS - Cent7.x. >>>>>> >>>>>> In which 2GB is allocated to oVirt. >>>>>> >>>>>> >>>>>> Version-Release number of selected component (if applicable): >>>>>> >>>>>> 4.2.2 >>>>>> >>>>>> >>>>>> How reproducible: >>>>>> >>>>>> If the number of concurrent requests are above 500, we are >>>>>> easily >>>>>> facing >>>>>> this issue. >>>>>> >>>>>> >>>>>> *Actual results:* >>>>>> >>>>>> SSL read failed (1) - closing connection >>>>>> 139620982339352:error: >>>>>> >>>>>> *Expected results:* >>>>>> >>>>>> Request success. >>>>>> >>>>>> >>>>>> Thanks, >>>>>> Hari >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users@ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
-- DISCLAIMER The information in this e-mail is confidential and may be subject to legal privilege. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you have received this communication in error, please address with the subject heading "Received in error," send to it@msystechnologies.com, then delete the e-mail and destroy any copies of it. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. The views, opinions, conclusions and other information expressed in this electronic mail and any attachments are not given or endorsed by the company unless otherwise indicated by an authorized representative independent of this message. MSys cannot guarantee that e-mail communications are secure or error-free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses, though all reasonable precautions have been taken to ensure no viruses are present in this e-mail. As our company cannot accept responsibility for any loss or damage arising from the use of this e-mail or attachments we recommend that you subject these to your virus checking procedures prior to use
Hi Juan, Could you help me with this question : 1) Is there a limitation in using the same token? Like one token can server only X number of request? On Thu, Mar 8, 2018 at 6:01 PM, Hari Prasanth Loganathan < hariprasanth.l@msystechnologies.com> wrote:
Hi Juan,
Now we got a clue, I try to hit 100 concurrent request in which 1 request got failed with the 503 error, I observed this in apache ssl_access.log
172.30.36.167 - - [08/Mar/2018:17:22:42 +0530] "GET /ovirt-engine/api/vms/ HTTP/1.1" 503 299
Is it thrown from Apache / ovirt engine?
On Thu, Mar 8, 2018 at 5:15 PM, Juan Hernández <jhernand@redhat.com> wrote:
I think the configuration is good. There may be some connections that the server is closing before the client expected, but that is normal, in my opinion.
On 03/08/2018 12:34 PM, Hari Prasanth Loganathan wrote:
This is the only error message we received from ab.
I googled it and found that it is due to the connection drop. So It would be Great, If you could check my Apache server configuration I shared in the thread and let me know your thoughts on this.
Thanks, Hari
On Thu, Mar 8, 2018 at 4:56 PM, Juan Hernández <jhernand@redhat.com> wrote:
But other than those SSL error messages, are the connections really
failing? Can you share the results reported by "ab"?
On 03/08/2018 12:16 PM, Hari Prasanth Loganathan wrote:
No Juan, It is not working with any benchmark / application tool. It
fails with the same error SSL handshake failed (5).
Could you let me know the configuration of Apache web server is correct?
Thanks, Hari
On Thu, Mar 8, 2018 at 1:08 AM, Juan Hernández <jhernand@redhat.com> wrote:
If you are still having problems I am inclined to think that it is a
client issue. For example, I'd try to remove the "-k" option from the "ab" command. If you use keep alive the server may decide anyhow to close the connection after certain number of requests, even if the client asks to keep it alive. Some clients don't handle that perfectly, "ab" may have that problem. If that makes the SSL error messages disappear then I think you can safely ignore them, and restore the "-k" option, if you want.
On 03/07/2018 07:30 PM, Hari Prasanth Loganathan wrote:
Thanks Juan for your response. Appreciate it.
> But for some reason still, I am facing the same SSL handshake failed > (5). > Could you please check this configuration and let me know the issue > in > my > ovirt engine environment. > > *Configuration of Apache server:* > > > 1) httpd version, > > # httpd -v > Server version: Apache/2.4.6 (CentOS) > Server built: Oct 19 2017 20:39:16 > > 2) I checked the status using the following command, > > # systemctl status httpd -l > ● httpd.service - The Apache HTTP Server > Loaded: loaded (/usr/lib/systemd/system/httpd.service; > enabled; > vendor > preset: disabled) > Active: active (running) since Wed 2018-03-07 23:46:32 IST; > 1min > 55s > ago > Docs: man:httpd(8) > man:apachectl(8) > Process: 4351 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, > status=0/SUCCESS) > Main PID: 4359 (httpd) > Status: "Total requests: 264; Current requests/sec: 0.1; > Current > traffic: 204 B/sec" > CGroup: /system.slice/httpd.service > ├─4359 /usr/sbin/httpd -DFOREGROUND > ├─4360 /usr/sbin/httpd -DFOREGROUND > ├─4362 /usr/sbin/httpd -DFOREGROUND > ├─5100 /usr/sbin/httpd -DFOREGROUND > ├─5386 /usr/sbin/httpd -DFOREGROUND > ├─5415 /usr/sbin/httpd -DFOREGROUND > └─5416 /usr/sbin/httpd -DFOREGROUND > > 3) Since the httpd is pointing to the path : > /usr/lib/systemd/system/httpd.service > > vi /usr/lib/systemd/system/httpd.service > > [Unit] > Description=The Apache HTTP Server > After=network.target remote-fs.target nss-lookup.target > Documentation=man:httpd(8) > Documentation=man:apachectl(8) > > [Service] > Type=notify > EnvironmentFile=/etc/sysconfig/httpd > ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND > ExecReload=/usr/sbin/httpd $OPTIONS -k graceful > ExecStop=/bin/kill -WINCH ${MAINPID} > # We want systemd to give httpd some time to finish gracefully, but > still > want > # it to kill httpd after TimeoutStopSec if something went wrong > during > the > # graceful stop. Normally, Systemd sends SIGTERM signal right after > the > # ExecStop, which would kill httpd. We are sending useless SIGCONT > here > to > give > # httpd time to finish. > KillSignal=SIGCONT > PrivateTmp=true > > [Install] > WantedBy=multi-user.target > > > 4) As per the above command I found the env file is available > '/etc/sysconfig/httpd' > > vi /etc/sysconfig/httpd > > # > # This file can be used to set additional environment variables for > # the httpd process, or pass additional options to the httpd > # executable. > # > # Note: With previous versions of httpd, the MPM could be changed by > # editing an "HTTPD" variable here. With the current version, that > # variable is now ignored. The MPM is a loadable module, and the > # choice of MPM can be changed by editing the configuration file > /etc/httpd/conf.modules.d/00-mpm.conf > # > > # > # To pass additional options (for instance, -D definitions) to the > # httpd binary at startup, set OPTIONS here. > # > #OPTIONS= > > # > # This setting ensures the httpd process is started in the "C" locale > # by default. (Some modules will not behave correctly if > # case-sensitive string comparisons are performed in a different > # locale.) > # > LANG=C > > > 5) As per the above command, I found that the conf fileis available > in > the > path : /etc/httpd/conf.modules.d/00-mpm.conf > > vi /etc/httpd/conf.modules.d/00-mpm.conf > > # Select the MPM module which should be used by uncommenting exactly > # one of the following LoadModule lines: > > # prefork MPM: Implements a non-threaded, pre-forking web server > # See: http://httpd.apache.org/docs/2.4/mod/prefork.html > #LoadModule mpm_prefork_module modules/mod_mpm_prefork.so > > # worker MPM: Multi-Processing Module implementing a hybrid > # multi-threaded multi-process web server > # See: http://httpd.apache.org/docs/2.4/mod/worker.html > # > LoadModule mpm_worker_module modules/mod_mpm_worker.so > > # event MPM: A variant of the worker MPM with the goal of consuming > # threads only for connections with active processing > # See: http://httpd.apache.org/docs/2.4/mod/event.html > # > #LoadModule mpm_event_module modules/mod_mpm_event.so > > <IfModule mpm_worker_module> > ServerLimit 1000 > MaxRequestWorkers 1000 > </IfModule> > > > > As per your comment, I enabled the 'LoadModule mpm_worker_module > modules/mod_mpm_worker.so' with the ServerLimit and > MaxRequestWorkers as > 1000 still I am facing the issue for the following command in apache > benchmark test. > > Completed 100 requests > SSL handshake failed (5). > SSL handshake failed (5). > SSL handshake failed (5). > SSL handshake failed (5). > SSL handshake failed (5). > SSL handshake failed (5). > > > NOTE : It always scales when I have concurrent request below 400 > > What is wrong in this apache configuration, why SSL handshake is > failing > for concurrent request above 400 ? > > Thanks, > Hari > > > > > > > On Wed, Mar 7, 2018 at 9:20 PM, Juan Hernández <jhernand@redhat.com> > wrote: > > It means that with the default configuration the Apache web server > can't > > serve more than 256 concurrent connections. This applies to any >> application >> that uses Apache as the web frontend, not just to oVirt. If you >> want to >> change that you have to change the MaxRequestWorkers and ServerLimit >> parameters, as explained here: >> >> https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m >> axrequestworkers >> >> So, go to your oVirt engine machine and create a >> /etc/httpd/conf.d/my.conf >> file with this content: >> >> MaxRequestWorkers 1000 >> ServerLimit 1000 >> >> Then restart the Apache server: >> >> # systemctl restart httpd >> >> Then your web server should be able to handle 1000 concurrent >> requests, >> and you will probably start to find other limits, like the amount of >> memory >> and CPU that those 1000 Apache child processes will consume, the >> number >> of >> threads in the JBoss application server, the number of connections >> to >> the >> database, etc. >> >> Let me insist a bit that if you base your benchmark solely on the >> number >> of concurrent requests or connections that the server can handle you >> may >> end up with meaningless results, as a real world application >> can/should >> use >> the server much better than that. >> >> On 03/07/2018 04:33 PM, Hari Prasanth Loganathan wrote: >> >> With the default configuration of the web server it is impossible to >> >> handle >>> more than 256 *connections* simultaneously. I guess that "ab" is >>> opening a >>> connection for each concurrent request, so when you reach request >>> 257 >>> the >>> web server will just reject the connection, there is nothing that >>> the >>> JBoss >>> can do about it; you have to increase the number of connections >>> supported >>> by the web server. >>> >>> *So Does it mean that oVirt cannot serve more than 257 request? * >>> >>> My question is, If its possible How to scale this and what is the >>> configuration we need to change? >>> >>> Also, we are taking a benchmark in using oVirt, So I need to find >>> the >>> maximum possible oVirt request. So please let me know the >>> configuration >>> tuning for oVirt to achieve maximum concurrent request. >>> >>> Thanks, >>> Hari >>> >>> On Wed, Mar 7, 2018 at 7:25 PM, Juan Hernández < >>> jhernand@redhat.com> >>> wrote: >>> >>> With the default configuration of the web server it is impossible >>> to >>> >>> handle more than 256 *connections* simultaneously. I guess that >>> "ab" >>> >>>> is >>>> opening a connection for each concurrent request, so when you >>>> reach >>>> request >>>> 257 the web server will just reject the connection, there is >>>> nothing >>>> that >>>> the JBoss can do about it; you have to increase the number of >>>> connections >>>> supported by the web server. >>>> >>>> Or else you may want to re-consider why you want to use 1000 >>>> simultaneous >>>> connections. It may be OK for a performance test, but there are >>>> better >>>> ways >>>> to squeeze performance. For example, you could consider using HTTP >>>> pipelining, which is much more friendly for the server than so >>>> many >>>> connections. This is what we use when we need to send a large >>>> number >>>> of >>>> requests from other systems. There are examples of how to do that >>>> with >>>> the >>>> Python and Ruby SDKs here: >>>> >>>> Python: >>>> >>>> https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/ >>>> examples/asynchronous_inventory.py >>>> >>>> Ruby: >>>> >>>> https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/ >>>> sdk/examples/asynchronous_inventory.rb >>>> >>>> >>>> On 03/07/2018 02:43 PM, Hari Prasanth Loganathan wrote: >>>> >>>> Hi Juan, >>>> >>>> >>>> Thanks for the response. >>>>> >>>>> I agree web server can handle only limited number of concurrent >>>>> requests. >>>>> But Why it is failing with SSL handshake failure for few >>>>> requests, >>>>> Can't >>>>> the JBOSS wait and serve the request? We can spare the delay but >>>>> not >>>>> with >>>>> the request fails. So Is there a configuration in oVirt which >>>>> can be >>>>> tuned >>>>> to achieve this? >>>>> >>>>> Thanks, >>>>> Hari >>>>> >>>>> On Wed, Mar 7, 2018 at 7:05 PM, Juan Hernández < >>>>> jhernand@redhat.com >>>>> >>>>>> >>>>>> wrote: >>>>> >>>>> The first thing you will need to change for such a test is the >>>>> number >>>>> of >>>>> >>>>> simultaneous connections accepted by the Apache web server: by >>>>> default >>>>> >>>>> the >>>>>> max is 256. See the Apache documentation here: >>>>>> >>>>>> https://httpd.apache.org/docs/2.4/mod/mpm_common.html#m >>>>>> axrequestworkers >>>>>> >>>>>> In addition I also suggest that you consider using the "worker" >>>>>> multi-processing module instead of the "prefork", as it usually >>>>>> works >>>>>> better when talking to a Java application server, because it >>>>>> re-uses >>>>>> connections better. >>>>>> >>>>>> On 03/07/2018 02:20 PM, Hari Prasanth Loganathan wrote: >>>>>> >>>>>> Hi Team, >>>>>> >>>>>> >>>>>> *Description of problem:* >>>>>> >>>>>>> >>>>>>> I am trying to achieve 1000 concurrent request to oVirt. What >>>>>>> are >>>>>>> the >>>>>>> tunable parameters to achieve this? >>>>>>> >>>>>>> I tried to perform the benchmarking for ovirt engine using >>>>>>> Apache >>>>>>> benchmark >>>>>>> using the same SSO token. >>>>>>> >>>>>>> ab -n 1000 -c 500 -k -H "accept: application/json" -H >>>>>>> "Authorization: >>>>>>> Bearer SSOTOKEN" https://172.30.56.70/ovirt-engine/ >>>>>>> <https://172.30.56.70/ovirt-engine/api/vms/5440271b-afb3-48b >>>>>>> b-9ff1-076fc07ebf50/statistics> >>>>>>> >>>>>>> When the number of concurrent request is 500, we are getting >>>>>>> more >>>>>>> than >>>>>>> 100 >>>>>>> failures with the following error, >>>>>>> >>>>>>> SSL read failed (1) - closing connection >>>>>>> 139620982339352:error: >>>>>>> >>>>>>> NOTE: It is scaling for concurrent request below 500. >>>>>>> >>>>>>> I used the profiler to get the memory and CPU and it seems very >>>>>>> less, >>>>>>> >>>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM >>>>>>> TIME+ >>>>>>> COMMAND >>>>>>> 30413 ovirt 20 0 4226664 882396 6776 S 126.0 23.0 >>>>>>> 27:48.53 >>>>>>> java >>>>>>> >>>>>>> Configuration of the machine in which Ovirt is deployed : >>>>>>> >>>>>>> RAM - 4GB, >>>>>>> Hard disk - 100GB, >>>>>>> core processor - 2, >>>>>>> OS - Cent7.x. >>>>>>> >>>>>>> In which 2GB is allocated to oVirt. >>>>>>> >>>>>>> >>>>>>> Version-Release number of selected component (if applicable): >>>>>>> >>>>>>> 4.2.2 >>>>>>> >>>>>>> >>>>>>> How reproducible: >>>>>>> >>>>>>> If the number of concurrent requests are above 500, we are >>>>>>> easily >>>>>>> facing >>>>>>> this issue. >>>>>>> >>>>>>> >>>>>>> *Actual results:* >>>>>>> >>>>>>> SSL read failed (1) - closing connection >>>>>>> 139620982339352:error: >>>>>>> >>>>>>> *Expected results:* >>>>>>> >>>>>>> Request success. >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Hari >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Users mailing list >>>>>>> Users@ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
-- DISCLAIMER The information in this e-mail is confidential and may be subject to legal privilege. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you have received this communication in error, please address with the subject heading "Received in error," send to it@msystechnologies.com, then delete the e-mail and destroy any copies of it. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. The views, opinions, conclusions and other information expressed in this electronic mail and any attachments are not given or endorsed by the company unless otherwise indicated by an authorized representative independent of this message. MSys cannot guarantee that e-mail communications are secure or error-free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses, though all reasonable precautions have been taken to ensure no viruses are present in this e-mail. As our company cannot accept responsibility for any loss or damage arising from the use of this e-mail or attachments we recommend that you subject these to your virus checking procedures prior to use
participants (3)
-
Hari Prasanth Loganathan -
Juan Hernández -
Moran Goldboim