[PATCH] [Kimchi] Use Websocket facilities from WoK
by dhbarboza82＠gmail.com
From: Daniel Henrique Barboza <danielhb(a)linux.vnet.ibm.com>
*** ATTENTION: REQUIRES '[WoK] Asynchronous UI notification implementation' PATCH ***
Daniel Henrique Barboza (1):
Use Websocket facilities from WoK
model/vms.py | 2 +-
root.py | 4 +-
websocket.py | 121 -----------------------------------------------------------
3 files changed, 2 insertions(+), 125 deletions(-)
delete mode 100644 websocket.py
--
2.9.3
7 years, 9 months
[PATCH] [Kimchi 0/2] Allow disks to update cache and io flags
by Ramon Medeiros
Ramon Medeiros (2):
Allow disks to update cache and io flags
Add tests to verify if cache and io of a disk can be changed
API.json | 11 ++++++++++-
docs/API.md | 3 +++
i18n.py | 1 -
model/vmstorages.py | 57 +++++++++++++++++++++++++++++++++--------------------
tests/test_model.py | 7 +++++++
xmlutils/disk.py | 12 ++++++++++-
6 files changed, 67 insertions(+), 24 deletions(-)
--
2.9.3
7 years, 9 months
[RFC] [Kimchi] #1092: Allow to set disk performance options per guest and template
by Ramon Medeiros
Propose: allow user to edit cache, io and bus for each disk
Make cache and io part was easy, but i got big trouble to change bus
Questions:
1) How we are going to change the bus? Got some issues aroud:
A) How to determine available bus? I know there is a function, but is it
works with vms with different buses?
B) From OS perspective? How can we deal? Only let the user choose the
driver on creation, not update?
7 years, 9 months
[PATCH] [WoK] Asynchronous UI notification implementation
by dhbarboza82＠gmail.com
From: Daniel Henrique Barboza <danielhb(a)linux.vnet.ibm.com>
This patch makes backend and UI changes to implement the asynchronous
UI notification in WoK.
- Backend:
A push server was implemented from scratch to manage the opened websocket
connections. The push server connects to the /run/woknotifications
UNIX socket and broadcasts all messages to all connections.
The websocket module is the same module that exists in the Kimchi
plug-in. The idea is to remove the module from Kimchi and make it
use the module from WoK. ws_proxy initialization was also added
in src/wok/server.py.
- Frontend:
In ui/js/wok.main.js two new functions were added to help the
usage of asynchronous notifications in the frontend. The idea:
a single websocket is opened per session. This opened websocket
will broadcast all incoming messages to all listeners registered.
Listeners can be added by the new wok.addNotificationListener()
method. This method will clean up any registered listener by
itself when the user changes tabs/URL.
The single websocket sends heartbeats to the backend side each
30 seconds. No reply from the backend is issued or expected. This
heartbeat is just a way to ensure that the browser does not
close the connection due to inactivity. This behavior varies from
browser to browser but this 30 second heartbeat is more than enough
to ensure that the websocket is kept alive.
- Working example in User Log:
A simple usage is provided in this patch. A change was made in
src/wok/reqlogger.py to send an asynchronous notification each
time a new log entry is created. In ui/js/wok.user-log.js a
websocket listener is added using wok.addNotificationListener()
and, for each message that indicates a new user log entry, a
refresh in the listing is issued.
Signed-off-by: Daniel Henrique Barboza <danielhb(a)linux.vnet.ibm.com>
---
src/wok/model/notifications.py | 18 +++++-
src/wok/pushserver.py | 132 +++++++++++++++++++++++++++++++++++++++++
src/wok/reqlogger.py | 7 ++-
src/wok/server.py | 8 ++-
src/wok/websocket.py | 123 ++++++++++++++++++++++++++++++++++++++
ui/js/src/wok.main.js | 38 ++++++++++++
ui/js/wok.user-log.js | 6 ++
7 files changed, 327 insertions(+), 5 deletions(-)
create mode 100644 src/wok/pushserver.py
create mode 100644 src/wok/websocket.py
diff --git a/src/wok/model/notifications.py b/src/wok/model/notifications.py
index bdb7c78..597eac5 100644
--- a/src/wok/model/notifications.py
+++ b/src/wok/model/notifications.py
@@ -1,7 +1,7 @@
#
# Project Wok
#
-# Copyright IBM Corp, 2016
+# Copyright IBM Corp, 2016-2017
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
@@ -19,12 +19,22 @@
from datetime import datetime
+from wok import config
from wok.exception import NotFoundError, OperationFailed
from wok.message import WokMessage
+from wok.pushserver import PushServer
from wok.utils import wok_log
notificationsStore = {}
+push_server = None
+
+
+def send_websocket_notification(message):
+ global push_server
+
+ if push_server:
+ push_server.send_notification(message)
def add_notification(code, args=None, plugin_name=None):
@@ -57,7 +67,11 @@ def del_notification(code):
class NotificationsModel(object):
def __init__(self, **kargs):
- pass
+ global push_server
+
+ test_mode = config.config.get('server', 'test').lower() == 'true'
+ if not test_mode:
+ push_server = PushServer()
def get_list(self):
global notificationsStore
diff --git a/src/wok/pushserver.py b/src/wok/pushserver.py
new file mode 100644
index 0000000..8993f00
--- /dev/null
+++ b/src/wok/pushserver.py
@@ -0,0 +1,132 @@
+#
+# Project Wok
+#
+# Copyright IBM Corp, 2017
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+#
+
+import cherrypy
+import os
+import select
+import socket
+import threading
+
+import websocket
+from utils import wok_log
+
+
+BASE_DIRECTORY = '/run'
+TOKEN_NAME = 'woknotifications'
+
+
+class PushServer(object):
+
+ def set_socket_file(self):
+ if not os.path.isdir(BASE_DIRECTORY):
+ try:
+ os.mkdir(BASE_DIRECTORY)
+ except OSError:
+ raise RuntimeError('PushServer base UNIX socket dir %s '
+ 'not found.' % BASE_DIRECTORY)
+
+ self.server_addr = os.path.join(BASE_DIRECTORY, TOKEN_NAME)
+
+ if os.path.exists(self.server_addr):
+ try:
+ os.remove(self.server_addr)
+ except:
+ raise RuntimeError('There is an existing connection in %s' %
+ self.server_addr)
+
+ def __init__(self):
+ self.set_socket_file()
+
+ websocket.add_proxy_token(TOKEN_NAME, self.server_addr, True)
+
+ self.connections = []
+
+ self.server_running = True
+ self.server_socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+ self.server_socket.setsockopt(socket.SOL_SOCKET,
+ socket.SO_REUSEADDR, 1)
+ self.server_socket.bind(self.server_addr)
+ self.server_socket.listen(10)
+ wok_log.info('Push server created on address %s' % self.server_addr)
+
+ self.connections.append(self.server_socket)
+ cherrypy.engine.subscribe('stop', self.close_server, 1)
+
+ server_loop = threading.Thread(target=self.listen)
+ server_loop.start()
+
+ def listen(self):
+ try:
+ while self.server_running:
+ read_ready, _, _ = select.select(self.connections,
+ [], [], 1)
+ for sock in read_ready:
+ if not self.server_running:
+ break
+
+ if sock == self.server_socket:
+
+ new_socket, addr = self.server_socket.accept()
+ self.connections.append(new_socket)
+ else:
+ try:
+ data = sock.recv(4096)
+ except:
+ try:
+ self.connections.remove(sock)
+ except ValueError:
+ pass
+
+ continue
+ if data and data == 'CLOSE':
+ sock.send('ACK')
+ try:
+ self.connections.remove(sock)
+ except ValueError:
+ pass
+ sock.close()
+
+ except Exception as e:
+ raise RuntimeError('Exception ocurred in listen() of pushserver '
+ 'module: %s' % e.message)
+
+ def send_notification(self, message):
+ for sock in self.connections:
+ if sock != self.server_socket:
+ try:
+ sock.send(message)
+ except IOError as e:
+ if 'Broken pipe' in str(e):
+ sock.close()
+ try:
+ self.connections.remove(sock)
+ except ValueError:
+ pass
+
+ def close_server(self):
+ try:
+ self.server_running = False
+ self.server_socket.shutdown(socket.SHUT_RDWR)
+ self.server_socket.close()
+ os.remove(self.server_addr)
+ except:
+ pass
+ finally:
+ cherrypy.engine.unsubscribe('stop', self.close_server)
diff --git a/src/wok/reqlogger.py b/src/wok/reqlogger.py
index 92e155d..1b774e2 100644
--- a/src/wok/reqlogger.py
+++ b/src/wok/reqlogger.py
@@ -1,7 +1,7 @@
#
# Project Wok
#
-# Copyright IBM Corp, 2016
+# Copyright IBM Corp, 2016-2017
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
@@ -34,6 +34,7 @@ from wok.auth import USER_NAME
from wok.config import get_log_download_path, paths
from wok.exception import InvalidParameter, OperationFailed
from wok.message import WokMessage
+from wok.model.notifications import send_websocket_notification
from wok.stringutils import ascii_dict
from wok.utils import remove_old_files
@@ -68,6 +69,8 @@ WOK_REQUEST_LOGGER = "wok_request_logger"
# AsyncTask handling
ASYNCTASK_REQUEST_METHOD = 'TASK'
+NEW_LOG_ENTRY_MESSAGE = 'new_log_entry'
+
def log_request(code, params, exception, method, status, app=None, user=None,
ip=None):
@@ -114,6 +117,8 @@ def log_request(code, params, exception, method, status, app=None, user=None,
ip=ip
).log()
+ send_websocket_notification(NEW_LOG_ENTRY_MESSAGE)
+
return log_id
diff --git a/src/wok/server.py b/src/wok/server.py
index fc2e167..2d823c9 100644
--- a/src/wok/server.py
+++ b/src/wok/server.py
@@ -25,8 +25,7 @@ import logging
import logging.handlers
import os
-from wok import auth
-from wok import config
+from wok import auth, config, websocket
from wok.config import config as configParser
from wok.config import WokConfig
from wok.control import sub_nodes
@@ -159,6 +158,11 @@ class Server(object):
cherrypy.tree.mount(WokRoot(model.Model(), dev_env),
options.server_root, self.configObj)
+ test_mode = config.config.get('server', 'test').lower() == 'true'
+ if not test_mode:
+ ws_proxy = websocket.new_ws_proxy()
+ cherrypy.engine.subscribe('exit', ws_proxy.terminate)
+
self._load_plugins()
cherrypy.lib.sessions.init()
diff --git a/src/wok/websocket.py b/src/wok/websocket.py
new file mode 100644
index 0000000..5d7fb91
--- /dev/null
+++ b/src/wok/websocket.py
@@ -0,0 +1,123 @@
+#!/usr/bin/env python2
+#
+# Project Wok
+#
+# Copyright IBM Corp, 2017
+#
+# Code derived from Project Kimchi
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+import base64
+import errno
+import os
+
+from multiprocessing import Process
+from websockify import WebSocketProxy
+
+from config import config, PluginPaths
+
+
+try:
+ from websockify.token_plugins import TokenFile
+ tokenFile = True
+except ImportError:
+ tokenFile = False
+
+try:
+ from websockify import ProxyRequestHandler as request_proxy
+except:
+ from websockify import WebSocketProxy as request_proxy
+
+
+WS_TOKENS_DIR = os.path.join(PluginPaths('kimchi').state_dir, 'ws-tokens')
+
+
+class CustomHandler(request_proxy):
+
+ def get_target(self, target_plugin, path):
+ if issubclass(CustomHandler, object):
+ target = super(CustomHandler, self).get_target(target_plugin,
+ path)
+ else:
+ target = request_proxy.get_target(self, target_plugin, path)
+
+ if target[0] == 'unix_socket':
+ try:
+ self.server.unix_target = target[1]
+ except:
+ self.unix_target = target[1]
+ else:
+ try:
+ self.server.unix_target = None
+ except:
+ self.unix_target = None
+ return target
+
+
+def new_ws_proxy():
+ try:
+ os.makedirs(WS_TOKENS_DIR, mode=0755)
+ except OSError as e:
+ if e.errno == errno.EEXIST:
+ pass
+
+ params = {'listen_host': '127.0.0.1',
+ 'listen_port': config.get('server', 'websockets_port'),
+ 'ssl_only': False}
+
+ # old websockify: do not use TokenFile
+ if not tokenFile:
+ params['target_cfg'] = WS_TOKENS_DIR
+
+ # websockify 0.7 and higher: use TokenFile
+ else:
+ params['token_plugin'] = TokenFile(src=WS_TOKENS_DIR)
+
+ def start_proxy():
+ try:
+ server = WebSocketProxy(RequestHandlerClass=CustomHandler,
+ **params)
+ except TypeError:
+ server = CustomHandler(**params)
+
+ server.start_server()
+
+ proc = Process(target=start_proxy)
+ proc.start()
+ return proc
+
+
+def add_proxy_token(name, port, is_unix_socket=False):
+ with open(os.path.join(WS_TOKENS_DIR, name), 'w') as f:
+ """
+ From python documentation base64.urlsafe_b64encode(s)
+ substitutes - instead of + and _ instead of / in the
+ standard Base64 alphabet, BUT the result can still
+ contain = which is not safe in a URL query component.
+ So remove it when needed as base64 can work well without it.
+ """
+ name = base64.urlsafe_b64encode(name).rstrip('=')
+ if is_unix_socket:
+ f.write('%s: unix_socket:%s' % (name.encode('utf-8'), port))
+ else:
+ f.write('%s: localhost:%s' % (name.encode('utf-8'), port))
+
+
+def remove_proxy_token(name):
+ try:
+ os.unlink(os.path.join(WS_TOKENS_DIR, name))
+ except OSError:
+ pass
diff --git a/ui/js/src/wok.main.js b/ui/js/src/wok.main.js
index 20c017e..f5031ce 100644
--- a/ui/js/src/wok.main.js
+++ b/ui/js/src/wok.main.js
@@ -29,6 +29,41 @@ wok.getConfig(function(result) {
wok.config = {};
});
+
+wok.notificationListeners = {};
+wok.addNotificationListener = function(name, func) {
+ wok.notificationListeners[name] = func;
+ $(window).one("hashchange", function() {
+ delete wok.notificationListeners[name];
+ });
+};
+
+wok.notificationsWebSocket = undefined;
+wok.startNotificationWebSocket = function () {
+ var addr = window.location.hostname + ':' + window.location.port;
+ var token = wok.urlSafeB64Encode('woknotifications').replace(/=*$/g, "");
+ var url = 'wss://' + addr + '/websockify?token=' + token;
+ wok.notificationsWebSocket = new WebSocket(url, ['base64']);
+
+ wok.notificationsWebSocket.onmessage = function(event) {
+ var message = window.atob(event.data);
+ for (name in wok.notificationListeners) {
+ func = wok.notificationListeners[name];
+ func(message);
+ }
+ };
+
+ sessionStorage.setItem('wokNotificationWebSocket', 'true');
+ var heartbeat = setInterval(function() {
+ wok.notificationsWebSocket.send(window.btoa('heartbeat'));
+ }, 30000);
+
+ wok.notificationsWebSocket.onclose = function() {
+ clearInterval(heartbeat);
+ };
+};
+
+
wok.main = function() {
wok.isLoggingOut = false;
wok.popable();
@@ -395,6 +430,9 @@ wok.main = function() {
// Set handler for help button
$('#btn-help').on('click', wok.openHelp);
+
+ // start WebSocket
+ wok.startNotificationWebSocket();
};
var initUI = function() {
diff --git a/ui/js/wok.user-log.js b/ui/js/wok.user-log.js
index 0e8fb09..083b6c3 100644
--- a/ui/js/wok.user-log.js
+++ b/ui/js/wok.user-log.js
@@ -153,6 +153,12 @@ wok.initUserLogContent = function() {
$("#user-log-grid").bootgrid("search");
wok.initUserLogConfigGridData();
});
+
+ wok.addNotificationListener('userlog', function(message) {
+ if (message === 'new_log_entry') {
+ $("#refresh-button").click();
+ }
+ });
};
wok.initUserLogWindow = function() {
--
2.9.3
7 years, 9 months
[PATCH V2] [Kimchi 0/2] Run wok tests without administration permissions
by Lucio Correia
- Updated test case instead of deleting it.
Lucio Correia (2):
Fix tests to run without proxy
Fix storage volume test to run without nginx
tests/test_model_storagevolume.py | 15 ++++++++++-----
tests/test_rest.py | 16 ++++++++--------
2 files changed, 18 insertions(+), 13 deletions(-)
--
2.7.4
7 years, 9 months
[PATCH V2] [Wok] Do not require admin permissions to run Wok tests
by Lucio Correia
- Applied review comments
- Rebase with master
- Replaced server 'test' option usage with new parameter 'no_proxy' due to
latest commits on master
Lucio Correia (1):
Do not use proxy when running tests
src/wok/server.py | 3 ++-
tests/utils.py | 17 ++++++++++++-----
2 files changed, 14 insertions(+), 6 deletions(-)
--
2.7.4
7 years, 9 months
[PATCH] [Wok] Bug fix #203: set default loglevel to INFO
by Ramon Medeiros
Signed-off-by: Ramon Medeiros <ramonn(a)linux.vnet.ibm.com>
---
src/wok.conf.in | 2 +-
src/wok/config.py.in | 2 +-
src/wok/server.py | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/src/wok.conf.in b/src/wok.conf.in
index 3806609..1ebdacf 100644
--- a/src/wok.conf.in
+++ b/src/wok.conf.in
@@ -37,7 +37,7 @@
#log_dir = @localstatedir@/log/wok
# Logging level: debug, info, warning, error or critical
-#log_level = debug
+#log_level = info
[authentication]
# Authentication method, available option: pam, ldap.
diff --git a/src/wok/config.py.in b/src/wok/config.py.in
index 8782a5f..97776dd 100644
--- a/src/wok/config.py.in
+++ b/src/wok/config.py.in
@@ -28,7 +28,7 @@ __version__ = "@wokversion@"
__release__ = "@wokrelease@"
CACHEEXPIRES = 31536000 # expires is one year.
-DEFAULT_LOG_LEVEL = "debug"
+DEFAULT_LOG_LEVEL = "info"
FONT_FILES = {'fontawesome': ['fontawesome-webfont.ttf'],
'opensans': ['OpenSans-ExtraBoldItalic.ttf',
diff --git a/src/wok/server.py b/src/wok/server.py
index fc2e167..7133233 100644
--- a/src/wok/server.py
+++ b/src/wok/server.py
@@ -103,7 +103,7 @@ class Server(object):
cherrypy.log.access_file = options.access_log
cherrypy.log.error_file = options.error_log
- logLevel = LOGGING_LEVEL.get(options.log_level, logging.DEBUG)
+ logLevel = LOGGING_LEVEL.get(options.log_level, logging.INFO)
dev_env = options.environment != 'production'
# Enable cherrypy screen logging if running environment
--
2.9.3
7 years, 9 months
vmtemplate logic while validating disk parameters
by Ramon Medeiros
Hi,
just looking at kimchi/vmtemplate.py, i saw this confusing logic:
101 basic_disk = ['index', 'format', 'pool', 'size']
102 basic_path_disk = ['index', 'format', 'path', 'size']
103 ro_disk = ['index', 'format', 'pool', 'volume']
104 base_disk = ['index', 'base', 'pool', 'size', 'format']
105 base_path_disk = ['index', 'base', 'path', 'size', 'format']
148 if ((keys != sorted(basic_disk)) and
149 (keys != sorted(ro_disk)) and
150 (keys != sorted(base_disk))):
151 # Addition check required only on s390x
152 if not is_s390x() or (keys !=
sorted(basic_path_disk)):
153 raise MissingParameter('KCHTMPL0028E')
The code is trying to validate if all fields are present, based on
combinations of parameters. I want to add more parameters, that are
non-optional. Is it good just add it or it's worth to rethink this logic?
What about removing this, since some validations already take place at
API.json?
7 years, 10 months
