[Kimchi-devel] [PATCH] [Wok] Use system's nginx proxy service.
Daniel Henrique Barboza
danielhb at linux.vnet.ibm.com
Mon Jul 11 12:24:18 UTC 2016
On 07/11/2016 09:10 AM, Paulo Ricardo Paz Vital wrote:
> On Jul 11 08:49AM, Daniel Henrique Barboza wrote:
>>
>> On 07/11/2016 08:05 AM, Paulo Ricardo Paz Vital wrote:
>>> On Jul 08 06:15PM, Daniel Henrique Barboza wrote:
>>>> make check-local errors found:
>>>>
>>>> [danielhb at arthas wok_all_plugins]$ sudo make check-local
>>>> PYTHONPATH=src contrib/check_i18n.py src/wok/plugins/*/i18n.py
>>>> src/wok/i18n.py
>>>> Checking for invalid i18n string...
>>>> Checking for invalid i18n string successfully
>>>> ./src/wok/proxy.py:29: 'subprocess' imported but unused
>>>> Makefile:930: recipe for target 'check-local' failed
>>>> make: *** [check-local] Error 1
>>>> [danielhb at arthas wok_all_plugins]$
>>>>
>>> Ouch!!! My check-local passed but I'm going to fix this.
>>>
>>>> I wasn't able to start wok after applying this patch. The error given is:
>>>>
>>>>
>>>> [danielhb at arthas wok_all_plugins]$ sudo src/wokd
>>>> [sudo] password for danielhb:
>>>> WOKPROXY0001E: WOKPROXY0001E
>>>> Traceback (most recent call last):
>>>> File "src/wokd", line 106, in <module>
>>>> sys.exit(main(sys.argv[1:]))
>>>> File "src/wokd", line 103, in main
>>>> wok.server.main(options)
>>>> File "/home/danielhb/kimchi/wok_all_plugins/src/wok/server.py", line 246,
>>>> in main
>>>> srv = Server(options)
>>>> File "/home/danielhb/kimchi/wok_all_plugins/src/wok/server.py", line 77,
>>>> in __init__
>>>> start_proxy(options)
>>>> File "/home/danielhb/kimchi/wok_all_plugins/src/wok/proxy.py", line 132,
>>>> in start_proxy
>>>> raise OperationFailed('WOKPROXY0001E', {'error': error})
>>>> wok.exception.OperationFailed: WOKPROXY0001E: WOKPROXY0001E
>>>> [danielhb at arthas wok_all_plugins]$
>>>>
>>> It's expected. My patch is restarting the nginx.service using systemctl
>>> command (since all supported distros today, use Systemd as default
>>> system) by the run_command(). So, I check if retcode is different of
>>> zero and raise this error.
>>>
>>>> I think it's because the nginx service isn't running in the host. However,
>>>> when trying to start the nginx service I got the following error:
>>>>
>>>> [danielhb at arthas conf.d]$ sudo systemctl start nginx
>>>> Job for nginx.service failed because the control process exited with error
>>>> code. See "systemctl status nginx.service" and "journalctl -xe" for details.
>>>>
>>>> [danielhb at arthas conf.d]$ sudo systemctl status nginx -l
>>>> ● nginx.service - The nginx HTTP and reverse proxy server
>>>> Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor
>>>> preset: disabled)
>>>> Active: failed (Result: exit-code) since Fri 2016-07-08 18:07:48 BRT; 11s
>>>> ago
>>>> Process: 18946 ExecStartPre=/usr/sbin/nginx -t (code=exited,
>>>> status=1/FAILURE)
>>>> Process: 18937 ExecStartPre=/usr/bin/rm -f /run/nginx.pid (code=exited,
>>>> status=0/SUCCESS)
>>>>
>>>> Jul 08 18:07:48 arthas.ltc.br.ibm.com systemd[1]: Starting The nginx HTTP
>>>> and reverse proxy server...
>>>> Jul 08 18:07:48 arthas.ltc.br.ibm.com nginx[18946]: nginx: [emerg] open()
>>>> "/etc/nginx/conf.d/wok.conf" failed (13: Permission denied) in
>>>> /etc/nginx/nginx.conf:33
>>>> Jul 08 18:07:48 arthas.ltc.br.ibm.com nginx[18946]: nginx: configuration
>>>> file /etc/nginx/nginx.conf test failed
>>>> Jul 08 18:07:48 arthas.ltc.br.ibm.com systemd[1]: nginx.service: Control
>>>> process exited, code=exited status=1
>>>> Jul 08 18:07:48 arthas.ltc.br.ibm.com systemd[1]: Failed to start The nginx
>>>> HTTP and reverse proxy server.
>>>> Jul 08 18:07:48 arthas.ltc.br.ibm.com systemd[1]: nginx.service: Unit
>>>> entered failed state.
>>>> Jul 08 18:07:48 arthas.ltc.br.ibm.com systemd[1]: nginx.service: Failed with
>>>> result 'exit-code'.
>>>> [danielhb at arthas conf.d]$
>>>>
>>>>
>>>> Says that it couldn't open /etc/nginx/conf.d/wok.conf. These are the
>>>> permissions:
>>>>
>>>> [danielhb at arthas conf.d]$ ls -lah
>>>> total 12K
>>>> drwxr-xr-x. 2 root root 4.0K Jul 8 18:08 .
>>>> drwxr-xr-x. 4 root root 4.0K Jun 11 09:17 ..
>>>> lrwxrwxrwx. 1 root root 56 Jul 8 18:08 wok.conf ->
>>>> /home/danielhb/kimchi/wok_all_plugins/src/nginx/wok.conf
>>>> -rw-r--r--. 1 root root 3.6K Jun 21 11:48 wok.conf.in
>>>> [danielhb at arthas conf.d]$
>>>> [danielhb at arthas conf.d]$ ls -lah
>>>> /home/danielhb/kimchi/wok_all_plugins/src/nginx/wok.conf
>>>> -rw-r--r--. 1 root root 3.2K Jul 8 18:08
>>>> /home/danielhb/kimchi/wok_all_plugins/src/nginx/wok.conf
>>>> [danielhb at arthas conf.d]$
>>>> [danielhb at arthas conf.d]$
>>>>
>>>>
>>>>
>>>> A quick search on the internet mention that SELinux might be on play
>>>> here, preventing the nginx service from reading the file. I didn't
>>>> tried to turn off SELinux to see if it works but I can try later. I've found
>>>> posts in Ubuntu forums mentioning that the file should be owned to the user
>>>> 'www-data' or something like that. Haven't tried that either.
>>>>
>>>> I'll try this patch in other systems I have access to see if the
>>>> results are the same. There's a chance that this error is specific
>>>> to my env.
>>>>
>>>>
>>> Yeah, the "failed (13: Permission denied)" error is a SELinux error, but
>>> in my tests on two Fedora 24 systems it was solved by the following
>>> command in console (I forgot comment this when submitted the patch):
>>>
>>> $ sudo restorecon -R -v /etc/nginx/conf.d/wok.conf
>>>
>>> In addition, it's necessary follow the SELinux steps reported on
>>> https://github.com/kimchi-project/wok/blob/master/docs/troubleshooting.md#selinux
>>>
>>> Something important to note, is that you're running from a NOT INSTALLED
>>> WOK then it's necessary create a symbolic link in /etc/nginx/conf.d/
>>> pointing to wok.conf file created from the path you're executing wokd. If
>>> you install wok, and execute from the installed system, the file is
>>> created if no problems.
>>>
>>> This symbolic link is necessary due to errors in nginx.service startup
>>> related to not able to open the *.perm and *.cert files in the path from
>>> the executed wokd.
>> I believe we can add these instructions of how to run WoK from source
>> in the README.
>>
> May be I was not clear. My patch already handle this, by checking if the
> running wokd is installed or not, and if not, it creates the symbolic link,
> since the nginx config file is always created when we start wokd.
Ok, I mentioned because I read
"Something important to note, is that you're running from a
NOT INSTALLED WOK then it's necessary create a symbolic link in
/etc/nginx/conf.d/ pointing to wok.conf file created from the path
you're executing wokd."
and thought that this was a manual step. If it's being taken care of
by the code then there's no need to add it to the README.
>
> I believe we can add the restorecon command in the SELinux section of
> Wok troubleshooting page.
Agree.
>
>>>> Last but not the least, there are a bunch of unit tests failing here
>>>> (FAILED (errors=7)) but I think they are related to the problem
>>>> I've described above (nginx not running).
>>>>
>>> I'm going to check these error also.
>>>
>>>> Daniel
>>>>
>>>> On 07/08/2016 01:26 PM, pvital at linux.vnet.ibm.com wrote:
>>>>> From: Paulo Vital <pvital at linux.vnet.ibm.com>
>>>>>
>>>>> This patch removes the code that executes a dedicated nginx proxy, making Wok
>>>>> to use the system's nginx service. This is a requirement to make Wok acceptable
>>>>> in community repositories.
>>>>>
>>>>> It also make sure that a Wok executed from path different than installed (from
>>>>> a cloned and built source code, for example) will create a symbolic link in
>>>>> system's nginx config dir to the running configuration file.
>>>>>
>>>>> This patch solves part of issue #25
>>>>>
>>>>> Signed-off-by: Paulo Vital <pvital at linux.vnet.ibm.com>
>>>>> ---
>>>>> src/nginx/wok.conf.in | 95 +++++++++++++++++++++------------------------------
>>>>> src/wok/config.py.in | 6 +++-
>>>>> src/wok/i18n.py | 2 ++
>>>>> src/wok/proxy.py | 27 +++++++++------
>>>>> src/wok/server.py | 6 +---
>>>>> 5 files changed, 63 insertions(+), 73 deletions(-)
>>>>>
>>>>> diff --git a/src/nginx/wok.conf.in b/src/nginx/wok.conf.in
>>>>> index cb05e4d..823d94d 100644
>>>>> --- a/src/nginx/wok.conf.in
>>>>> +++ b/src/nginx/wok.conf.in
>>>>> @@ -22,71 +22,54 @@
>>>>> # This is a template file to be used to generate a nginx
>>>>> # proxy config file at wokd script.
>>>>>
>>>>> -user ${user};
>>>>> -worker_processes 1;
>>>>> +client_max_body_size ${max_body_size}k;
>>>>>
>>>>> -error_log /var/log/nginx/error.log;
>>>>> +# Timeout set to 10 minutes to avoid the 504 Gateway Timeout
>>>>> +# when Wok is processing a request.
>>>>> +proxy_connect_timeout 600;
>>>>> +proxy_send_timeout 600;
>>>>> +proxy_read_timeout 600;
>>>>> +send_timeout 600;
>>>>>
>>>>> -events {
>>>>> - worker_connections 1024;
>>>>> +map $http_upgrade $connection_upgrade {
>>>>> + default upgrade;
>>>>> + '' close;
>>>>> }
>>>>>
>>>>> -http {
>>>>> - log_format main '$remote_addr - $remote_user [$time_local] "$request" '
>>>>> - '$status $body_bytes_sent "$http_referer" '
>>>>> - '"$http_user_agent" "$http_x_forwarded_for"';
>>>>> -
>>>>> - access_log /var/log/nginx/access.log main;
>>>>> - sendfile on;
>>>>> +upstream websocket {
>>>>> + server 127.0.0.1:${websockets_port};
>>>>> +}
>>>>>
>>>>> - client_max_body_size ${max_body_size}k;
>>>>> +server {
>>>>> + listen ${host_addr}:${proxy_ssl_port} ssl;
>>>>>
>>>>> - # Timeout set to 10 minutes to avoid the 504 Gateway Timeout
>>>>> - # when Wok is processing a request.
>>>>> - proxy_connect_timeout 600;
>>>>> - proxy_send_timeout 600;
>>>>> - proxy_read_timeout 600;
>>>>> - send_timeout 600;
>>>>> + ssl_certificate ${cert_pem};
>>>>> + ssl_certificate_key ${cert_key};
>>>>> + ssl_protocols TLSv1.1 TLSv1.2;
>>>>> + ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:@STRENGTH';
>>>>> + ssl_prefer_server_ciphers on;
>>>>> + ssl_dhparam ${dhparams_pem};
>>>>> + ssl_session_timeout ${session_timeout}m;
>>>>>
>>>>> - map $http_upgrade $connection_upgrade {
>>>>> - default upgrade;
>>>>> - '' close;
>>>>> - }
>>>>> + add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
>>>>> + add_header X-Frame-Options DENY;
>>>>> + add_header X-Content-Type-Options nosniff;
>>>>> + add_header X-XSS-Protection "1; mode=block";
>>>>>
>>>>> - upstream websocket {
>>>>> - server 127.0.0.1:${websockets_port};
>>>>> + location / {
>>>>> + proxy_pass http://127.0.0.1:${cherrypy_port};
>>>>> + proxy_set_header Host $host;
>>>>> + proxy_set_header X-Real-IP $remote_addr;
>>>>> + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>>>>> + proxy_redirect http://127.0.0.1:${cherrypy_port}/ https://$host:${proxy_ssl_port}/;
>>>>> }
>>>>>
>>>>> - server {
>>>>> - listen ${host_addr}:${proxy_ssl_port} ssl;
>>>>> -
>>>>> - ssl_certificate ${cert_pem};
>>>>> - ssl_certificate_key ${cert_key};
>>>>> - ssl_protocols TLSv1.1 TLSv1.2;
>>>>> - ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:@STRENGTH';
>>>>> - ssl_prefer_server_ciphers on;
>>>>> - ssl_dhparam ${dhparams_pem};
>>>>> - ssl_session_timeout ${session_timeout}m;
>>>>> -
>>>>> - add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
>>>>> - add_header X-Frame-Options DENY;
>>>>> - add_header X-Content-Type-Options nosniff;
>>>>> - add_header X-XSS-Protection "1; mode=block";
>>>>> -
>>>>> - location / {
>>>>> - proxy_pass http://127.0.0.1:${cherrypy_port};
>>>>> - proxy_set_header Host $host;
>>>>> - proxy_set_header X-Real-IP $remote_addr;
>>>>> - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>>>>> - proxy_redirect http://127.0.0.1:${cherrypy_port}/ https://$host:${proxy_ssl_port}/;
>>>>> - }
>>>>> -
>>>>> - location /websockify {
>>>>> - proxy_pass http://websocket;
>>>>> - proxy_http_version 1.1;
>>>>> - proxy_set_header Upgrade $http_upgrade;
>>>>> - proxy_set_header Connection $connection_upgrade;
>>>>> - }
>>>>> + location /websockify {
>>>>> + proxy_pass http://websocket;
>>>>> + proxy_http_version 1.1;
>>>>> + proxy_set_header Upgrade $http_upgrade;
>>>>> + proxy_set_header Connection $connection_upgrade;
>>>>> }
>>>>> - ${http_config}
>>>>> }
>>>>> +
>>>>> +${http_config}
>>>>> diff --git a/src/wok/config.py.in b/src/wok/config.py.in
>>>>> index 1ca6f73..d0bca9b 100644
>>>>> --- a/src/wok/config.py.in
>>>>> +++ b/src/wok/config.py.in
>>>>> @@ -76,9 +76,10 @@ class Paths(object):
>>>>> self.prefix = self.get_prefix()
>>>>> self.installed = (self.prefix == '@pkgdatadir@')
>>>>> self.ui_dir = self.add_prefix('ui')
>>>>> + self.sys_nginx_conf_dir = '@sysconfdir@/nginx/conf.d'
>>>>>
>>>>> if self.installed:
>>>>> - self.nginx_conf_dir = '@sysconfdir@/nginx/conf.d'
>>>>> + self.nginx_conf_dir = self.sys_nginx_conf_dir
>>>>> self.state_dir = '@localstatedir@/lib/wok'
>>>>> self.log_dir = '@localstatedir@/log/wok'
>>>>> self.conf_dir = '@sysconfdir@/wok'
>>>>> @@ -119,6 +120,9 @@ class Paths(object):
>>>>> def get_template_path(self, resource):
>>>>> return os.path.join(self.ui_dir, 'pages/%s.tmpl' % resource)
>>>>>
>>>>> + def is_wok_installed(self):
>>>>> + return self.installed
>>>>> +
>>>>>
>>>>> paths = Paths()
>>>>>
>>>>> diff --git a/src/wok/i18n.py b/src/wok/i18n.py
>>>>> index d6cb17c..33107ee 100644
>>>>> --- a/src/wok/i18n.py
>>>>> +++ b/src/wok/i18n.py
>>>>> @@ -55,6 +55,8 @@ messages = {
>>>>> "WOKUTILS0004E": _("Invalid data value '%(value)s'"),
>>>>> "WOKUTILS0005E": _("Invalid data unit '%(unit)s'"),
>>>>>
>>>>> + "WOKPROXY0001E": _("Unable to (re)start system's nginx.service. Details: '%(error)s'"),
>>>>> +
>>>>> # These messages (ending with L) are for user log purposes
>>>>> "WOKCOL0001L": _("Request made on collection"),
>>>>> "WOKRES0001L": _("Request made on resource"),
>>>>> diff --git a/src/wok/proxy.py b/src/wok/proxy.py
>>>>> index a74e88a..ab959e1 100644
>>>>> --- a/src/wok/proxy.py
>>>>> +++ b/src/wok/proxy.py
>>>>> @@ -31,6 +31,8 @@ from string import Template
>>>>>
>>>>> from wok import sslcert
>>>>> from wok.config import paths
>>>>> +from wok.exception import OperationFailed
>>>>> +from wok.utils import run_command
>>>>>
>>>>>
>>>>> HTTP_CONFIG = """
>>>>> @@ -110,18 +112,21 @@ def _create_proxy_config(options):
>>>>> config_file.write(data)
>>>>> config_file.close()
>>>>>
>>>>> + # If not running from the installed path (from a cloned and builded source
>>>>> + # code), create a symbolic link in system's dir to prevent errors on read
>>>>> + # SSL certifications.
>>>>> + if not paths.is_wok_installed():
>>>>> + dst = os.path.join(paths.sys_nginx_conf_dir, "wok.conf")
>>>>> + if os.path.isfile(dst):
>>>>> + os.remove(dst)
>>>>> + os.symlink(os.path.join(nginx_config_dir, "wok.conf"), dst)
>>>>> +
>>>>>
>>>>> def start_proxy(options):
>>>>> """Start nginx reverse proxy."""
>>>>> _create_proxy_config(options)
>>>>> - nginx_config_dir = paths.nginx_conf_dir
>>>>> - config_file = "%s/wok.conf" % nginx_config_dir
>>>>> - cmd = ['nginx', '-c', config_file]
>>>>> - subprocess.call(cmd)
>>>>> -
>>>>> -
>>>>> -def terminate_proxy():
>>>>> - """Stop nginx process."""
>>>>> - config_file = "%s/wok.conf" % paths.nginx_conf_dir
>>>>> - term_proxy_cmd = ['nginx', '-s', 'stop', '-c', config_file]
>>>>> - subprocess.call(term_proxy_cmd)
>>>>> + # Restart system's nginx service to reload wok configuration
>>>>> + cmd = ['systemctl', 'restart', 'nginx.service']
>>>>> + output, error, retcode = run_command(cmd, silent=True)
>>>>> + if retcode != 0:
>>>>> + raise OperationFailed('WOKPROXY0001E', {'error': error})
>>>>> diff --git a/src/wok/server.py b/src/wok/server.py
>>>>> index 8a02596..b1185e3 100644
>>>>> --- a/src/wok/server.py
>>>>> +++ b/src/wok/server.py
>>>>> @@ -33,7 +33,7 @@ from wok.config import config as configParser
>>>>> from wok.config import paths, PluginConfig, WokConfig
>>>>> from wok.control import sub_nodes
>>>>> from wok.model import model
>>>>> -from wok.proxy import start_proxy, terminate_proxy
>>>>> +from wok.proxy import start_proxy
>>>>> from wok.reqlogger import RequestLogger
>>>>> from wok.root import WokRoot
>>>>> from wok.safewatchedfilehandler import SafeWatchedFileHandler
>>>>> @@ -180,10 +180,6 @@ class Server(object):
>>>>> self.app = cherrypy.tree.mount(WokRoot(model_instance, dev_env),
>>>>> config=self.configObj)
>>>>> self._load_plugins(options)
>>>>> -
>>>>> - # Terminate proxy when cherrypy server is terminated
>>>>> - cherrypy.engine.subscribe('exit', terminate_proxy)
>>>>> -
>>>>> cherrypy.lib.sessions.init()
>>>>>
>>>>> def _load_plugins(self, options):
>>>>> --
>>>>> 2.7.4
>>>>>
>>>>> _______________________________________________
>>>>> Kimchi-devel mailing list
>>>>> Kimchi-devel at ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>>>> _______________________________________________
>>>> Kimchi-devel mailing list
>>>> Kimchi-devel at ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>> _______________________________________________
>> Kimchi-devel mailing list
>> Kimchi-devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
More information about the Kimchi-devel
mailing list