Re: [Kimchi-devel] [PATCH] [Wok] Add federation feature to Wok

It was primarily implemented on Kimchi project. But it makes more sense to be on Wok since the idea is to allow user add/select on which server Wok should manage. For example, the user could have a Wok installation on server A to manage server B. Signed-off-by: Aline Manera <alinefm(a)linux.vnet.ibm.com&gt; --- docs/API/config.md | 2 ++ docs/API/peers.md | 20 +++++++++++ docs/README-federation.md | 60 ++++++++++++++++++++++++++++++++ docs/wokd.8.in | 6 +++- src/wok.conf.in | 4 +++ src/wok/config.py.in | 1 + src/wok/control/peers.py | 30 ++++++++++++++++ src/wok/model/config.py | 1 + src/wok/model/peers.py | 74 +++++++++++++++++++++++++++++++++++++++ src/wokd.in | 5 +++ tests/test_api.py | 6 +++- tests/test_config_model.py | 2 +- tests/test_server_root.py | 6 ++-- ui/js/src/wok.api.js | 14 ++++++++ ui/js/wok.peers.js | 75 ++++++++++++++++++++++++++++++++++++++++ ui/pages/tabs/settings.html.tmpl | 34 +++++++++++++++++- 16 files changed, 333 insertions(+), 7 deletions(-) create mode 100644 docs/API/peers.md create mode 100644 docs/README-federation.md create mode 100644 src/wok/control/peers.py create mode 100644 src/wok/model/peers.py create mode 100644 ui/js/wok.peers.js diff --git a/docs/API/config.md b/docs/API/config.md index d1d1007..0e47f5f 100644 --- a/docs/API/config.md +++ b/docs/API/config.md @@ -12,6 +12,8 @@ Contains information about the application environment and configuration. * proxy_port: SSL port to list on * websockets_port: Port for websocket proxy to listen on * auth: Authentication method used to log in to Wok + * server_root: Relative path to Wok server. No value is specified by default, ie, Wok will run on '/' + * federation: 'on' if federation feature is enabled, 'off' otherwise. * version: Wok version * **POST**: *See Task Actions* diff --git a/docs/API/peers.md b/docs/API/peers.md new file mode 100644 index 0000000..fa5cf41 --- /dev/null +++ b/docs/API/peers.md @@ -0,0 +1,20 @@ +## REST API Specification for Peers + +### Collection: Peers + +**URI:** /peers + +Return a list of Wok peers in the same network. +(It uses openSLP for discovering) + +**Methods:** + +* **GET**: Retrieve a list peers URLs. + +#### Examples +GET /peers +[ + https://wok-peer0:8001, + https://wok-peer1:8001, + https://wok-peer2:8001, +] diff --git a/docs/README-federation.md b/docs/README-federation.md new file mode 100644 index 0000000..18e19c7 --- /dev/null +++ b/docs/README-federation.md @@ -0,0 +1,60 @@ +Wok Project - Federation Feature +=================================== + +Federation feature is a mechanism to discover Wok peers in the same +network. It uses openSLP tool (http://www.openslp.org/) to register and find Wok +servers. + +By default this feature is disabled on Wok as it is not critical for KVM +virtualization and requires additional software installation. + +To enable it, do the following: + +1. Install openslp and openslp-server rpm packages, + or install slpd and slptool deb packages. + +2. openSLP uses port 427 (UDP) and port 427 (TCP) so make sure to open those + ports in your firewall configuration + + For system using firewalld, do: + sudo firewall-cmd --permanent --add-port=427/udp + sudo firewall-cmd --permanent --add-port=427/tcp + sudo firewall-cmd --reload + + For openSUSE systems, do: + sudo /sbin/SuSEfirewall2 open EXT TCP 427 + sudo /sbin/SuSEfirewall2 open EXT UDP 427 + + For system using iptables, do: + sudo iptables -A INPUT -p tcp --dport 427 -j ACCEPT + sudo iptables -A INPUT -p udp --dport 427 -j ACCEPT + +3. In addition to the openSLP ports, you also need to allow multicast in the + firewall configuration + + For system using firewalld, do: + sudo firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -s <subnet> -j ACCEPT + + For openSUSE systems, do: + Add the subnet to the trusted networks listed on FW_TRUSTED_NETS in + /etc/sysconfig/SuSEfirewall2 file. + Make sure to restart /sbin/SuSEfirewall2 after modifying /etc/sysconfig/SuSEfirewall2 + + For system using iptables, do: + sudo iptables -A INPUT -s <subnet> -j ACCEPT + +4. Start slpd service and make sure it is up while running Wok + sudo service slpd start + +5. Enable federation on Wok by editing the /etc/wok/wok.conf file: + + federation = on + +6. Then start Wok service + sudo service wokd start + +The Wok server will be registered on openSLP on server starting up and will +be found by other Wok peers (with federation feature enabled) in the same +network. + +Enjoy! diff --git a/docs/wokd.8.in b/docs/wokd.8.in index 217d9f6..09b5847 100644 --- a/docs/wokd.8.in +++ b/docs/wokd.8.in @@ -7,7 +7,7 @@ wokd - Launch Wok web server [\fB--cherrypy_port\fP \fICHERRYPY_PORT\fP] [\fB--websockets_port\fP \fIWEBSOCKETS_PORT\fP] [\fB--session_timeout\fP \fISESSION_TIMEOUT\fP] [\fB--log_level\fP \fILOG_LEVEL\fP] [\fB--log_dir\fP \fILOG_DIR\fP] [\fB--environment\fP \fIENV\fP] -[\fB--server_root\fP \fISERVER_ROOT\fP] [\fB--test\fP] +[\fB--server_root\fP \fISERVER_ROOT\fP] [\fB--federation\fP \fIfederation\fP] [\fB--test\fP] .SH DESCRIPTION \fBWok\fP is a cherrypy-based web framework with HTML5 support originated from Kimchi. It can be extended by plugins which expose functionality through REST APIs. @@ -55,6 +55,10 @@ Check cherrypy documentation for more details (default \fIproduction\fP). \fB\-\-server_root\fP [\fISERVER_ROOT\fP] Relative path to Wok server. No value is specified by default, ie, Wok will run on '/'. .TP +\fB\-\-federation\fP [\fIon\fP | \fIoff\fP] +Register and discover Wok peers in the same network using OpenSLP. Check +below the \fBREAD-federation\fP for more details (default \fIoff\fP). +.TP \fB\-\-test\fP Run Wok on a mock version that does not affect the system. For testing proposals. It depends on how plugins implements the mock environment as well. diff --git a/src/wok.conf.in b/src/wok.conf.in index 1ebdacf..1ed6310 100644 --- a/src/wok.conf.in +++ b/src/wok.conf.in @@ -27,6 +27,10 @@ # uncomment the following: #server_root=/wok +# Federation feature: register Wok server on openSLP and discover peers +# in the same network. Check README-federation for more details. +#federation = off + [logging] # Log directory diff --git a/src/wok/config.py.in b/src/wok/config.py.in index bbf2c09..39d7ad2 100644 --- a/src/wok/config.py.in +++ b/src/wok/config.py.in @@ -277,6 +277,7 @@ def _get_config(): config.set("server", "environment", "production") config.set('server', 'max_body_size', '4*1024*1024') config.set("server", "server_root", "") + config.set("server", "federation", "off") config.set("server", "test", "") config.add_section("authentication") config.set("authentication", "method", "pam") diff --git a/src/wok/control/peers.py b/src/wok/control/peers.py new file mode 100644 index 0000000..2a14815 --- /dev/null +++ b/src/wok/control/peers.py @@ -0,0 +1,30 @@ +# +# Project Wok +# +# Copyright IBM Corp, 2017 +# +# Code derived from Kimchi Project +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +from wok.control.base import SimpleCollection +from wok.control.utils import UrlSubNode + + +@UrlSubNode("peers", True) +class Peers(SimpleCollection): + def __init__(self, model): + super(Peers, self).__init__(model) + self.admin_methods = ['GET'] diff --git a/src/wok/model/config.py b/src/wok/model/config.py index b69f2dd..887ceba 100644 --- a/src/wok/model/config.py +++ b/src/wok/model/config.py @@ -34,6 +34,7 @@ class ConfigModel(object): 'websockets_port': config.get('server', 'websockets_port'), 'auth': config.get('authentication', 'method'), 'server_root': config.get('server', 'server_root'), + 'federation': config.get('server', 'federation'), 'version': get_version()} def reload(self, name): diff --git a/src/wok/model/peers.py b/src/wok/model/peers.py new file mode 100644 index 0000000..02afd59 --- /dev/null +++ b/src/wok/model/peers.py @@ -0,0 +1,74 @@ +# +# Project Wok +# +# Copyright IBM Corp, 2017 +# +# Code derived from Kimchi Project +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +import cherrypy +import re +import socket + +from wok.config import config +from wok.utils import run_command, wok_log + + +class PeersModel(object): + def __init__(self, **kargs): + # check federation feature is enabled on Wok server + if not config.get('server', 'federation') == 'off': + return + + # register server on openslp + hostname = socket.getfqdn() + port = config.get("server", "proxy_port") + self.url = hostname + ":" + port + + cmd = ["slptool", "register", + "service:wokd://%s" % self.url] + out, error, ret = run_command(cmd) + if out and len(out) != 0: + wok_log.error("Unable to register server on openSLP." + " Details: %s" % out) + cherrypy.engine.subscribe('exit', self._peer_deregister) + + def _peer_deregister(self): + cmd = ["slptool", "deregister", + "service:wokd://%s" % self.url] + out, error, ret = run_command(cmd) + if out and len(out) != 0: + wok_log.error("Unable to deregister server on openSLP." + " Details: %s" % out) + + def get_list(self): + # check federation feature is enabled on Wok server + if not config.get('server', 'federation') == 'off': + return [] + + cmd = ["slptool", "findsrvs", "service:wokd"] + out, error, ret = run_command(cmd) + if ret != 0: + return [] + + peers = [] + for server in out.strip().split("\n"): + match = re.match("service:wokd://(.*?),.*", server) + peer = match.group(1) + if peer != self.url: + peers.append("https://" + peer) + + return peers diff --git a/src/wokd.in b/src/wokd.in index 29586a0..dfe85a8 100644 --- a/src/wokd.in +++ b/src/wokd.in @@ -48,6 +48,7 @@ def main(options): websockets_port = config.config.get("server", "websockets_port") session_timeout = config.config.get("server", "session_timeout") runningEnv = config.config.get("server", "environment") + federation = config.config.get("server", "federation") server_root = config.config.get("server", "server_root") logDir = config.config.get("logging", "log_dir") logLevel = config.config.get("logging", "log_level") @@ -71,6 +72,10 @@ def main(options): help="Running environment of wok server") parser.add_option('--server_root', type="string", default=server_root, help="Relative path to server") + parser.add_option('--federation', default=federation, + help="Register and discover Wok peers in the same " + "network using openSLP. Check README-federation for" + " more details.") parser.add_option('--test', action='store_true', help="Run server in mock model") (options, args) = parser.parse_args() diff --git a/tests/test_api.py b/tests/test_api.py index 6fbee75..f978c93 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -49,11 +49,15 @@ class APITests(unittest.TestCase): def setUp(self): self.request = partial(utils.request) + def test_peers(self): + resp = self.request('/peers').read() + self.assertEquals([], json.loads(resp)) + def test_config(self): resp = self.request('/config').read() conf = json.loads(resp) keys = ["auth", "proxy_port", "websockets_port", "version", - "server_root"] + "server_root", "federation"] self.assertEquals(sorted(keys), sorted(conf.keys())) def test_config_plugins(self): diff --git a/tests/test_config_model.py b/tests/test_config_model.py index f8b0848..565b4df 100644 --- a/tests/test_config_model.py +++ b/tests/test_config_model.py @@ -30,7 +30,7 @@ class ConfigModelTests(unittest.TestCase): config = inst.config_lookup('') self.assertItemsEqual( ['proxy_port', 'websockets_port', 'auth', - 'server_root', 'version'], + 'server_root', 'version', 'federation'], config.keys() ) diff --git a/tests/test_server_root.py b/tests/test_server_root.py index e95a13b..89f34d3 100644 --- a/tests/test_server_root.py +++ b/tests/test_server_root.py @@ -1,7 +1,7 @@ # # Project Wok # -# Copyright IBM Corp, 2016 +# Copyright IBM Corp, 2016-2017 # # This library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public @@ -49,7 +49,7 @@ class ServerRootTests(unittest.TestCase): # check if server_root in config is the same used to start server resp = request(server_root + '/config').read() conf = json.loads(resp) - self.assertEquals(len(conf), 5) + self.assertEquals(len(conf), 6) def test_development_env(self): """ @@ -61,4 +61,4 @@ class ServerRootTests(unittest.TestCase): # check if server_root in config is the same used to start server resp = request(server_root + '/config').read() conf = json.loads(resp) - self.assertEquals(len(conf), 5) + self.assertEquals(len(conf), 6) diff --git a/ui/js/src/wok.api.js b/ui/js/src/wok.api.js index 06b97aa..4e68eee 100644 --- a/ui/js/src/wok.api.js +++ b/ui/js/src/wok.api.js @@ -71,6 +71,20 @@ var wok = { }); }, + getPeers: function(suc, err) { + wok.requestJSON({ + url: 'peers', + type: 'GET', + contentType: 'application/json', + dataType: 'json', + resend: true, + success: suc, + error: err ? err : function(data) { + wok.message.error(data.responseJSON.reason); + } + }); + }, + getNotifications: function (suc, err) { wok.requestJSON({ url: 'notifications', diff --git a/ui/js/wok.peers.js b/ui/js/wok.peers.js new file mode 100644 index 0000000..67e58b0 --- /dev/null +++ b/ui/js/wok.peers.js @@ -0,0 +1,75 @@ +/* + * Project Wok + * + * Copyright IBM Corp, 2017 + * + * Code derived from Kimchi Project + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +wok.initPeers = function() { + var peersDatatableTable; + var peers = new Array(); + + var peersDatatable = function(nwConfigDataSet) { + peersDatatableTable = $('#peers-list').DataTable({ + "processing": true, + "data": peers, + "language": { + "emptyTable": i18n['WOKSETT0010M'] + }, + "order": [], + "paging": false, + "dom": '<"row"<"col-sm-12"t>>', + "scrollY": "269px", + "scrollCollapse": true, + "columnDefs": [{ + "targets": 0, + "searchable": false, + "orderable": false, + "width": "100%", + "className": "tabular-data", + "render": function(data, type, full, meta) { + return '<a href="' + data + '" target="_blank">' + data + '</a>'; + } + }], + "initComplete": function(settings, json) { + $('#peers-content-area > .wok-mask').addClass('hidden'); + } + }); + }; + + var getPeers = function() { + wok.getPeers(function(result) { + peers.length = 0; + for (var i = 0; i < result.length; i++) { + var tempArr = []; + tempArr.push(result[i]); + peers.push(tempArr); + } + peersDatatable(peers); + }, function(err) { + wok.message.error(err.responseJSON.reason, '#peers-alert-container', true); + }); + }; + getPeers(); + +} + +if (wok.config.federation == 'on') { + $("#peers-accordion").removeClass('hidden'); + wok.initPeers(); +} else { + $("#peers-accordion").addClass('hidden'); +} diff --git a/ui/pages/tabs/settings.html.tmpl b/ui/pages/tabs/settings.html.tmpl index ccc5b01..5f52839 100644 --- a/ui/pages/tabs/settings.html.tmpl +++ b/ui/pages/tabs/settings.html.tmpl @@ -27,6 +27,7 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA <head> <link rel="stylesheet" type="text/css" href="$href('css/settings.css')"> <script type="text/javascript" src="$href('js/wok.settings.js')"></script> + <script type="text/javascript" src="$href('js/wok.peers.js')"></script> <script type="text/javascript" src="$href('js/wok.bootgrid.js')"></script> </head> @@ -34,7 +35,7 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA <div id="wok-root-container" class="wok"> <div class="container"> <div id="wokSettings" class="wok-settings"> - <!-- Plugins Management --> + <!-- Plugins Management --> <div class="panel-group accordion" id="plugins-mgmt-accordion" role="tablist" aria-multiselectable="false"> <h3> <a role="button" data-toggle="collapse" data-parent="#plugin-mgmt-accordion" href="#plugins-mgmt-content-area" aria-expanded="true" aria-controls="plugins-mgmt-content-area" class=""> @@ -65,6 +66,37 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA </div> </div> <!-- --> + <!-- Peers --> + <div class='panel-group federation-enabled accordion hidden' id='peers-accordion' role='tablist' aria-multiselectable='true'> + <h3> + <a role='button' data-toggle='collapse' data-parent='#peers-accordion' href='#peers-content-area' aria-expanded='false' aria-controls='peers-content-area' class=''> + <span class='accordion-icon'></span><span class='accordion-text' id='#peers-title'>$_("Peers")</span> + </a> + </h3> + <div id='peers-content-area' class='panel-collapse collapse' role='tabpanel' aria-labelledby='peers-title'> + <div id='peers-alert-container'></div> + <div class='row'> + <div class='col-sm-12'> + <table id='peers-list' class='table table-striped' cellspacing='0' width='100%'> + <thead> + <tr> + <th><span class='sr-only'>$_("Peers")</span></th> + </tr> + </thead> + </table> + </div> + </div> + <div class='wok-mask' role='presentation'> + <div class='wok-mask-loader-container'> + <div class='wok-mask-loading'> + <div class='wok-mask-loading-icon'></div> + <div class='wok-mask-loading-text'>$_("Loading...")</div> + </div> + </div> + </div> + </div> + </div> + <!-- --> </div> </div> </div>