April 2015 - Kimchi-devel - oVirt List Archives

RFC - Guest memory usage
by Daniel Henrique Barboza 27 Oct '15

27 Oct '15

I was looking at the following item in the backlog: Guests Stats: Display memory utilization (use virt-df or virt-top or ...) If I understood it right, the idea here is to show the -inner- memory allocation of the guest. If you have a VM with 4Gb of RAM running an Ubuntu, we want to know how much memory the Ubuntu OS and its processes are using. I've done an investigation and I haven't found any tool to accomplish this. "virt-top", "virsh dommemstat" and the libvirt API retrieves the information of the memory usage of the guest relative to the host. In the example mentioned before, supposing that the host has 64Gb of RAM, all these tools would show that the VM is using 12% of the host RAM. They do not dive in the VM and shows the actual mem usage of the Ubuntu and its processes running there. Haven't found anything useful in other MLs and forums. The common answer is 'run top in a terminal inside the VM', which of course does not suit us. My question is: any thoughts about how we can implement this feature? Because I am starting to think that, in the end, this kind of info is strict to the guest OS and can't be polled from the outside. Thanks!

5 5

[RFC] Split Kimchi into Base Framework and Virtualization layers
by Lucio Correia 05 May '15

05 May '15

Hi, Gustavo (on cc) and me have been analyzing how to split Kimchi Virtualization layer form its Base web framework. We would like to leverage all infrastructure kimchi base framework provides (URI routing, error handling, i18n, authentication, API validation, plugin tabs loading) in other projects, and contribute this back to kimchi-ginger project. The general idea for the web framework is that it provides basic UI with login and tabs menu, and each plugin installed extends the menu with some tabs. We started by making kimchi a plugin, like ginger is. We are way advanced on that and already have most of kimchi's UI running as a plugin over an almost fully running base web framework. Ginger needs minimal changes, mainly on import statements. We can provide changes needed back to ginger too. Currently, we are splitting kimchi specific stuff like this way: (after renaming the dir src/kimchi to src/backend *) * we are using the name "backend" just for disambiguation from virtualization layer (plugins/kimchi). We will need to define a better name for that basic framework. One suggestion is to call it "wok", the base where kimchi is cooked. {src/backend => plugins/kimchi}/API.json | 0 {src/backend/control/vm => plugins/kimchi/control}/Makefile.am | 7 +- {src/backend/control/vm => plugins/kimchi/control}/__init__.py | 2 +- {src/backend => plugins/kimchi}/control/config.py | 0 {src/backend => plugins/kimchi}/control/cpuinfo.py | 0 {src/backend => plugins/kimchi}/control/debugreports.py | 0 {src/backend => plugins/kimchi}/control/groups.py | 0 {src/backend => plugins/kimchi}/control/host.py | 0 {src/backend => plugins/kimchi}/control/interfaces.py | 0 {src/backend => plugins/kimchi}/control/networks.py | 0 {src/backend => plugins/kimchi}/control/peers.py | 0 {src/backend => plugins/kimchi}/control/storagepools.py | 0 {src/backend => plugins/kimchi}/control/storageservers.py | 0 {src/backend => plugins/kimchi}/control/storagevolumes.py | 0 {src/backend => plugins/kimchi}/control/templates.py | 0 {src/backend => plugins/kimchi}/control/users.py | 0 {src/backend => plugins/kimchi}/control/vm/Makefile.am | 0 {src/backend => plugins/kimchi}/control/vm/__init__.py | 0 {src/backend => plugins/kimchi}/control/vm/hostdevs.py | 0 {src/backend => plugins/kimchi}/control/vm/ifaces.py | 0 {src/backend => plugins/kimchi}/control/vm/snapshots.py | 0 {src/backend => plugins/kimchi}/control/vm/storages.py | 0 {src/backend => plugins/kimchi}/control/vms.py | 0 {src/backend => plugins/kimchi}/disks.py | 0 {src/backend => plugins/kimchi}/imageinfo.py | 0 {src/backend => plugins/kimchi}/iscsi.py | 0 {src/backend => plugins/kimchi}/isoinfo.py | 0 {src/backend => plugins/kimchi}/kvmusertests.py | 0 {src/backend => plugins/kimchi}/mockmodel.py | 0 {src/backend/control/vm => plugins/kimchi/model}/Makefile.am | 7 +- {src/backend/control/vm => plugins/kimchi/model}/__init__.py | 8 --- {src/backend => plugins/kimchi}/model/config.py | 0 {src/backend => plugins/kimchi}/model/cpuinfo.py | 0 {src/backend => plugins/kimchi}/model/debugreports.py | 0 {src/backend => plugins/kimchi}/model/diskutils.py | 0 {src/backend => plugins/kimchi}/model/featuretests.py | 0 {src/backend => plugins/kimchi}/model/groups.py | 0 {src/backend => plugins/kimchi}/model/host.py | 0 {src/backend => plugins/kimchi}/model/hostdev.py | 0 {src/backend => plugins/kimchi}/model/interfaces.py | 0 {src/backend => plugins/kimchi}/model/libvirtconnection.py | 0 {src/backend => plugins/kimchi}/model/libvirtstoragepool.py | 0 plugins/kimchi/model/model.py | 93 ++++++++++++++++++++++++++++ {src/backend => plugins/kimchi}/model/networks.py | 0 {src/backend => plugins/kimchi}/model/peers.py | 0 {src/backend => plugins/kimchi}/model/storagepools.py | 0 {src/backend => plugins/kimchi}/model/storageservers.py | 0 {src/backend => plugins/kimchi}/model/storagetargets.py | 0 {src/backend => plugins/kimchi}/model/storagevolumes.py | 0 {src/backend => plugins/kimchi}/model/templates.py | 0 {src/backend => plugins/kimchi}/model/users.py | 0 {src/backend => plugins/kimchi}/model/utils.py | 0 {src/backend => plugins/kimchi}/model/vmhostdevs.py | 0 {src/backend => plugins/kimchi}/model/vmifaces.py | 0 {src/backend => plugins/kimchi}/model/vms.py | 0 {src/backend => plugins/kimchi}/model/vmsnapshots.py | 0 {src/backend => plugins/kimchi}/model/vmstorages.py | 0 {src/backend => plugins/kimchi}/osinfo.py | 0 {src/backend => plugins/kimchi}/scan.py | 0 {config => plugins/kimchi/ui}/Makefile.am | 0 {config/ui => plugins/kimchi/ui/config}/Makefile.am | 0 config/ui/tabs.xml => plugins/kimchi/ui/config/tab-ext.xml | 0 {src/backend => plugins/kimchi}/vmtemplate.py | 0 The remaining generic stuff on the "backend" side, usable or already in use by other plugins, would be: backend |___model | |___model.py | |___plugins.py | |___tasks.py |___control | |___base.py | |___plugins.py | |___tasks.py | |___utils.py |___xmlutils | |___network.py | |___disk.py | |___utils.py | |___graphics.py | |___qemucmdline.py | |___interface.py |___asynctask.py |___auth.py |___basemodel.py |___cachebust.py |___distroloader.py |___exception.py |___i18n.py |___netinfo.py |___network.py |___objectstore.py |___proxy.py |___repositories.py |___rollbackcontext.py |___root.py |___screenshot.py |___server.py |___sslcert.py |___swupdate.py |___template.py |___utils.py |___vnc.py We don't want to make this a too long e-mail and lose the focus on the idea. We have further technical details to discuss in a follow up. Please let us know your thoughts on that, any directions and requirements you have on this, and how could we proceed with that plan. -- Lucio Correia Software Engineer IBM LTC Brazil

2 3

[kimchi-devel][PATCHv4 0/7] Upload storage volume
by lvroyce＠linux.vnet.ibm.com 04 May '15

04 May '15

From: Royce Lv <lvroyce(a)linux.vnet.ibm.com> How to upload: POST /storagepools/<pool-name>/storagevolumes/ {'capacity':size-of-the-volume,'name':name-of-the-volume,'format':'raw'} PUT /storagepools/<pool-name>/storagevolumes/<volume-name> {'chunk':chunk-data,'chunk_size':size-of-chunk,'index':index-of-the-chunk-of-uploaded-file} PATCH 7/7 is a script against real model server, you can use it to verify the logic of upload. Royce Lv (7): Update docs and json schema of storage volume upload Update controller to make update accept formdata params Add lock facility for storage volume upload Update model for storage volume update Fix incomplete record when uploading update test case for storage volume upload A test script against real model server docs/API.md | 7 ++- src/kimchi/API.json | 22 ++++++++ src/kimchi/control/base.py | 6 +-- src/kimchi/i18n.py | 4 ++ src/kimchi/isoinfo.py | 5 +- src/kimchi/model/storagevolumes.py | 61 ++++++++++----------- src/kimchi/model/utils.py | 46 ++++++++++++++++ tests/test_api.py | 108 +++++++++++++++++++++++++++++++++++++ tests/test_model_storagevolume.py | 95 ++++++++++++++++++++++++-------- 9 files changed, 291 insertions(+), 63 deletions(-) create mode 100644 tests/test_api.py -- 2.1.0

3 15

[PATCH 0/5] Support vm name with special characters
by Ramon Medeiros 04 May '15

04 May '15

Ramon Medeiros (5): Create title tag Change the way that get_vm() and vm_list() search for VMs Use get_vm() instead of conn.lookupByName Encode vm_name to generate VNC token Reflect the unicode conversion on token creation src/kimchi/model/vms.py | 50 +++++++++++++++++++++++++++++++++--------- src/kimchi/model/vmstorages.py | 4 ++-- src/kimchi/vmtemplate.py | 4 +++- ui/js/src/kimchi.api.js | 4 ++-- 4 files changed, 47 insertions(+), 15 deletions(-) -- 2.1.0

2 11

[PATCH v2] Supress error messages while checking vm metadata
by Ramon Medeiros 04 May '15

04 May '15

Changes v2: Use parse functions present on model/utils.py The function FeatureTests.disable_libvirt_error_logging only removes the error message that will be displayed on the console. Libvirt also logs errors on messages log. To avoid flood messages log with metadata errors, it will be verified manually if the xml has the tag. Signed-off-by: Ramon Medeiros <ramonn(a)linux.vnet.ibm.com> --- src/kimchi/model/utils.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/kimchi/model/utils.py b/src/kimchi/model/utils.py index b2739b2..1b114f6 100644 --- a/src/kimchi/model/utils.py +++ b/src/kimchi/model/utils.py @@ -95,6 +95,9 @@ def _kimchi_set_metadata_node(dom, node): def libvirt_get_kimchi_metadata_node(dom, mode="current"): + if _kimchi_get_metadata_node(dom, "metadata") == None: + return + FeatureTests.disable_libvirt_error_logging() try: xml = dom.metadata(libvirt.VIR_DOMAIN_METADATA_ELEMENT, -- 2.1.0

2 1

[PATCH 0/3] Snapshot enhancements
by Crístian Deives 04 May '15

04 May '15

Crístian Deives (3): snapshot: Handle non-existing snapshots in mock lookup snapshot: Change the action 'revert' to a Task snapshot: Allow creating a snapshot of a domain in any state src/kimchi/control/vm/snapshots.py | 2 +- src/kimchi/i18n.py | 1 - src/kimchi/mockmodel.py | 14 ++++++++++++++ src/kimchi/model/vmsnapshots.py | 36 ++++++++++++++++++++++-------------- tests/test_model.py | 15 +++++++++++---- tests/test_rest.py | 13 +++++++++++++ 6 files changed, 61 insertions(+), 20 deletions(-) -- 2.1.0

3 9

[PATCH] Do not resolve hostname to IP in using remote ISOs
by Aline Manera 04 May '15

04 May '15

Due a limitation in old versions of qemu-io (qemu-io was not able to resolve hostnames preveting the ISO streaming support), Kimchi automatically resolved the hostname to IP to workaround this problem. As all the most used Linux distributions (RHEl7.1, Ubuntu 14.10, Fedora 21 and openSUSE 13.2) have a working version of qemu-io, Kimchi does not need to care about it anymore. It also fixes the issue #436 Signed-off-by: Aline Manera <alinefm(a)linux.vnet.ibm.com> Signed-off-by: Daniel Henrique Barboza <danielhb(a)linux.vnet.ibm.com> --- src/kimchi/model/config.py | 2 -- src/kimchi/model/featuretests.py | 26 -------------------------- src/kimchi/model/utils.py | 11 ----------- src/kimchi/model/vms.py | 1 - src/kimchi/model/vmstorages.py | 10 +++------- src/kimchi/vmtemplate.py | 12 ++---------- tests/test_model.py | 11 +---------- 7 files changed, 6 insertions(+), 67 deletions(-) diff --git a/src/kimchi/model/config.py b/src/kimchi/model/config.py index a417c9b..ca0a74b 100644 --- a/src/kimchi/model/config.py +++ b/src/kimchi/model/config.py @@ -51,7 +51,6 @@ class CapabilitiesModel(object): def __init__(self, **kargs): self.conn = kargs['conn'] self.qemu_stream = False - self.qemu_stream_dns = False self.libvirt_stream_protocols = [] self.fc_host_support = False self.metadata_support = False @@ -88,7 +87,6 @@ class CapabilitiesModel(object): kimchi_log.info("*** Running feature tests ***") conn = self.conn.get() self.qemu_stream = FeatureTests.qemu_supports_iso_stream() - self.qemu_stream_dns = FeatureTests.qemu_iso_stream_dns() self.nfs_target_probe = FeatureTests.libvirt_support_nfs_probe(conn) self.fc_host_support = FeatureTests.libvirt_support_fc_host(conn) self.metadata_support = FeatureTests.has_metadata_support(conn) diff --git a/src/kimchi/model/featuretests.py b/src/kimchi/model/featuretests.py index 8cbe31d..9400151 100644 --- a/src/kimchi/model/featuretests.py +++ b/src/kimchi/model/featuretests.py @@ -21,9 +21,7 @@ import cherrypy import libvirt import lxml.etree as ET import platform -import socket import subprocess -import threading from lxml.builder import E @@ -153,30 +151,6 @@ class FeatureTests(object): return len(stderr) == 0 @staticmethod - @servermethod - def qemu_iso_stream_dns(): - host = socket.getfqdn(cherrypy.server.socket_host) - port = cherrypy.server.socket_port - cmd = ["qemu-io", "-r", "http://%s:%d/images/icon-fedora.png" % - (host, port), "-c", "'read -v 0 512'"] - proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - - thread = threading.Thread(target=proc.communicate) - thread.start() - thread.join(5) - - if thread.is_alive(): - proc.kill() - thread.join() - return False - - if proc.returncode != 0: - return False - - return True - - @staticmethod def libvirt_support_fc_host(conn): try: FeatureTests.disable_libvirt_error_logging() diff --git a/src/kimchi/model/utils.py b/src/kimchi/model/utils.py index b2739b2..0c9d519 100644 --- a/src/kimchi/model/utils.py +++ b/src/kimchi/model/utils.py @@ -18,8 +18,6 @@ # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA import libvirt -import socket -import urlparse from lxml import etree, objectify from lxml.builder import E, ElementMaker @@ -41,15 +39,6 @@ def get_vm_name(vm_name, t_name, name_list): raise OperationFailed("KCHUTILS0003E") -def check_remote_disk_path(path, qemu_stream_dns): - hostname = urlparse.urlparse(path).hostname - if hostname is not None and not qemu_stream_dns: - ip = socket.gethostbyname(hostname) - return path.replace(hostname, ip) - - return path - - def get_vm_config_flag(dom, mode="persistent"): # libvirt.VIR_DOMAIN_AFFECT_CURRENT is 0 # VIR_DOMAIN_AFFECT_LIVE is 1, VIR_DOMAIN_AFFECT_CONFIG is 2 diff --git a/src/kimchi/model/vms.py b/src/kimchi/model/vms.py index c8267c1..a6ca27b 100644 --- a/src/kimchi/model/vms.py +++ b/src/kimchi/model/vms.py @@ -125,7 +125,6 @@ class VMsModel(object): stream_protocols = self.caps.libvirt_stream_protocols xml = t.to_vm_xml(name, vm_uuid, libvirt_stream_protocols=stream_protocols, - qemu_stream_dns=self.caps.qemu_stream_dns, graphics=graphics, volumes=vol_list) diff --git a/src/kimchi/model/vmstorages.py b/src/kimchi/model/vmstorages.py index 37aca64..87c6b3d 100644 --- a/src/kimchi/model/vmstorages.py +++ b/src/kimchi/model/vmstorages.py @@ -1,7 +1,7 @@ # # Project Kimchi # -# Copyright IBM, Corp. 2014 +# Copyright IBM, Corp. 2014-2015 # # This library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public @@ -26,7 +26,7 @@ from kimchi.exception import OperationFailed from kimchi.model.config import CapabilitiesModel from kimchi.model.vms import DOM_STATE_MAP, VMModel from kimchi.model.storagevolumes import StorageVolumeModel -from kimchi.model.utils import check_remote_disk_path, get_vm_config_flag +from kimchi.model.utils import get_vm_config_flag from kimchi.osinfo import lookup from kimchi.model.diskutils import get_disk_ref_cnt, set_disk_ref_cnt from kimchi.utils import kimchi_log @@ -131,8 +131,6 @@ class VMStoragesModel(object): params['disk'] = vol_info['type'] params.update(self._get_available_bus_address(params['bus'], vm_name)) - params['path'] = check_remote_disk_path(params['path'], - self.caps.qemu_stream_dns) # Add device to VM dev, xml = get_disk_xml(params) @@ -215,9 +213,7 @@ class VMStorageModel(object): if dev_info['type'] != 'cdrom': raise InvalidOperation("KCHVMSTOR0006E") - params['path'] = check_remote_disk_path(params.get('path', ''), - self.caps.qemu_stream_dns) - + params['path'] = params.get('path', '') old_disk_path = dev_info['path'] new_disk_path = params['path'] if new_disk_path != old_disk_path: diff --git a/src/kimchi/vmtemplate.py b/src/kimchi/vmtemplate.py index ef97d83..e047228 100644 --- a/src/kimchi/vmtemplate.py +++ b/src/kimchi/vmtemplate.py @@ -18,7 +18,6 @@ # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA import os -import socket import stat import time import urlparse @@ -128,7 +127,7 @@ class VMTemplate(object): except IsoFormatError: raise InvalidParameter("KCHISO0001E", {'filename': iso}) - def _get_cdrom_xml(self, libvirt_stream_protocols, qemu_stream_dns): + def _get_cdrom_xml(self, libvirt_stream_protocols): if 'cdrom' not in self.info: return '' @@ -139,11 +138,6 @@ class VMTemplate(object): params['index'] = self.info['cdrom_index'] params['path'] = self.info['cdrom'] - hostname = urlparse.urlparse(params['path']).hostname - if hostname is not None and not qemu_stream_dns: - ip = socket.gethostbyname(hostname) - params['path'] = params['path'].replace(hostname, ip) - if self.info.get('iso_stream', False): protocol = urlparse.urlparse(params['path']).scheme if protocol not in libvirt_stream_protocols: @@ -304,10 +298,8 @@ class VMTemplate(object): graphics.update(kwargs.get('graphics', {})) params['graphics'] = get_graphics_xml(graphics) - qemu_stream_dns = kwargs.get('qemu_stream_dns', False) libvirt_stream_protocols = kwargs.get('libvirt_stream_protocols', []) - cdrom_xml = self._get_cdrom_xml(libvirt_stream_protocols, - qemu_stream_dns) + cdrom_xml = self._get_cdrom_xml(libvirt_stream_protocols) if not urlparse.urlparse(self.info.get('cdrom', "")).scheme in \ libvirt_stream_protocols and \ diff --git a/tests/test_model.py b/tests/test_model.py index 210adfd..bd195b5 100644 --- a/tests/test_model.py +++ b/tests/test_model.py @@ -23,9 +23,7 @@ import os import pwd import re import shutil -import socket import time -import urlparse import unittest import uuid @@ -564,14 +562,7 @@ class ModelTests(unittest.TestCase): {'path': valid_remote_iso_path}) cdrom_info = inst.vmstorage_lookup(vm_name, cdrom_dev) cur_cdrom_path = re.sub(":80/", '/', cdrom_info['path']) - - # As Kimchi server is not running during this test case - # CapabilitiesModel.qemu_stream_dns will be always False - # so we need to convert the hostname to IP - output = urlparse.urlparse(valid_remote_iso_path) - hostname = socket.gethostbyname(output.hostname) - url = valid_remote_iso_path.replace(output.hostname, hostname) - self.assertEquals(url, cur_cdrom_path) + self.assertEquals(valid_remote_iso_path, cur_cdrom_path) @unittest.skipUnless(utils.running_as_root(), 'Must be run as root') def test_vm_storage_provisioning(self): -- 2.1.0

2 2

[PATCH 0/7] Make Template defaults configurable
by Aline Manera 30 Apr '15

30 Apr '15

Aline Manera (7): Remove useless variable in osinfo.py Merge common_spec wiht defaults value in osinfo.py Make Template defaults configurable Add libvirt-daemon-config-network package as Kimchi dependency Verify all networks set as Template defaults prior to server start up Create option to auto create ISO pool or not on server start up Verify storage pool set as Template default prior to server starts up contrib/DEBIAN/control.in | 1 + contrib/kimchi.spec.fedora.in | 3 ++ contrib/kimchi.spec.suse.in | 3 ++ docs/README.md | 14 +++--- src/Makefile.am | 2 +- src/kimchi.conf.in | 3 ++ src/kimchi/config.py.in | 1 + src/kimchi/model/model.py | 44 +---------------- src/kimchi/model/networks.py | 61 ++++++++++------------- src/kimchi/model/storagepools.py | 66 +++++++++++++++++++++++++ src/kimchi/model/templates.py | 2 - src/kimchi/osinfo.py | 101 ++++++++++++++++++++++++++++++--------- src/kimchid.in | 3 ++ src/template.conf | 47 ++++++++++++++++++ 14 files changed, 240 insertions(+), 111 deletions(-) create mode 100644 src/template.conf -- 2.1.0

2 15

[PATCH 0/6 v7] Asyn VM creation
by Aline Manera 30 Apr '15

30 Apr '15

V6 -> V7: - Update test cases - Update UI to reuse the clone box for pending VM creation Aline Manera (1): Create VMs Asynchronously: UI Christy Perez (5): Append clone to target_uri for vm clone task Tests for new clone target_uri UI changes for new clone target_uri Create VMs asynchronously: Backend Create VMs Asynchronously: Tests src/kimchi/control/vms.py | 4 +-- src/kimchi/model/vms.py | 33 ++++++++++++++++++----- tests/test_authorization.py | 25 +++++++++++------- tests/test_mockmodel.py | 14 +++++++--- tests/test_model.py | 49 +++++++++++++++++++++++----------- tests/test_rest.py | 60 +++++++++++++++++++++++++++++++----------- tests/test_server.py | 26 ++++++++---------- ui/css/theme-default/list.css | 8 +++--- ui/js/src/kimchi.guest_main.js | 32 ++++++++++++++++++---- ui/pages/guest.html.tmpl | 4 +-- ui/pages/i18n.json.tmpl | 1 + 11 files changed, 176 insertions(+), 80 deletions(-) -- 2.1.0

2 9

kimchi Chef cookbook
by Scott Hain 30 Apr '15

30 Apr '15

Hello, I wanted to share that I've put together a Chef cookbook for running a local dev/test instance of kimchi. I am using it for my own testing and I thought it might be useful to someone else. https://github.com/opscode-cookbooks/kimchi I created for internal testing, but it could easily be modified to set up hardware or other VMs (it may work already for these use cases, but I haven't tested them). Thanks! -- *Scott Hain* — Software Engineer II 360.360.7433 – shain(a)chef.io – *my: *Linkedin <http://www.linkedin.com/> Twitter <http://www.twitter.com/elejia> *CHEF* CHEF.IO <http://chef.io> TM *chef.io* <http://www.chef.io/> Blog <http://www.chef.io/blog/> Facebook <https://www.facebook.com/getchefdotcom> Twitter <https://twitter.com/chef> Youtube <https://www.youtube.com/getchef>

2 1