RFC - Guest memory usage
by Daniel Henrique Barboza
I was looking at the following item in the backlog:
Guests Stats: Display memory utilization (use virt-df or virt-top or ...)
If I understood it right, the idea here is to show the -inner- memory
allocation of the guest. If you have a VM with 4Gb of RAM running an
Ubuntu, we want to know how much memory the Ubuntu OS and its processes
are using.
I've done an investigation and I haven't found any tool to accomplish
this. "virt-top", "virsh dommemstat" and the libvirt API retrieves the
information of the memory usage of the guest relative to the host. In
the example mentioned before, supposing that the host has 64Gb of RAM,
all these tools would show that the VM is using 12% of the host RAM.
They do not dive in the VM and shows the actual mem usage of the Ubuntu
and its processes running there.
Haven't found anything useful in other MLs and forums. The common answer
is 'run top in a terminal inside the VM', which of course does not suit
us. My question is: any thoughts about how we can implement this
feature? Because I am starting to think that, in the end, this kind of
info is strict to the guest OS and can't be polled from the outside.
Thanks!
9 years, 2 months
[RFC] Split Kimchi into Base Framework and Virtualization layers
by Lucio Correia
Hi,
Gustavo (on cc) and me have been analyzing how to split Kimchi
Virtualization layer form its Base web framework. We would like to
leverage all infrastructure kimchi base framework provides (URI routing,
error handling, i18n, authentication, API validation, plugin tabs
loading) in other projects, and contribute this back to kimchi-ginger
project.
The general idea for the web framework is that it provides basic UI with
login and tabs menu, and each plugin installed extends the menu with
some tabs.
We started by making kimchi a plugin, like ginger is. We are way
advanced on that and already have most of kimchi's UI running as a
plugin over an almost fully running base web framework.
Ginger needs minimal changes, mainly on import statements. We can
provide changes needed back to ginger too.
Currently, we are splitting kimchi specific stuff like this way: (after
renaming the dir src/kimchi to src/backend *)
* we are using the name "backend" just for disambiguation from
virtualization layer (plugins/kimchi). We will need to define a better
name for that basic framework. One suggestion is to call it "wok", the
base where kimchi is cooked.
{src/backend => plugins/kimchi}/API.json | 0
{src/backend/control/vm => plugins/kimchi/control}/Makefile.am | 7 +-
{src/backend/control/vm => plugins/kimchi/control}/__init__.py | 2 +-
{src/backend => plugins/kimchi}/control/config.py | 0
{src/backend => plugins/kimchi}/control/cpuinfo.py | 0
{src/backend => plugins/kimchi}/control/debugreports.py | 0
{src/backend => plugins/kimchi}/control/groups.py | 0
{src/backend => plugins/kimchi}/control/host.py | 0
{src/backend => plugins/kimchi}/control/interfaces.py | 0
{src/backend => plugins/kimchi}/control/networks.py | 0
{src/backend => plugins/kimchi}/control/peers.py | 0
{src/backend => plugins/kimchi}/control/storagepools.py | 0
{src/backend => plugins/kimchi}/control/storageservers.py | 0
{src/backend => plugins/kimchi}/control/storagevolumes.py | 0
{src/backend => plugins/kimchi}/control/templates.py | 0
{src/backend => plugins/kimchi}/control/users.py | 0
{src/backend => plugins/kimchi}/control/vm/Makefile.am | 0
{src/backend => plugins/kimchi}/control/vm/__init__.py | 0
{src/backend => plugins/kimchi}/control/vm/hostdevs.py | 0
{src/backend => plugins/kimchi}/control/vm/ifaces.py | 0
{src/backend => plugins/kimchi}/control/vm/snapshots.py | 0
{src/backend => plugins/kimchi}/control/vm/storages.py | 0
{src/backend => plugins/kimchi}/control/vms.py | 0
{src/backend => plugins/kimchi}/disks.py | 0
{src/backend => plugins/kimchi}/imageinfo.py | 0
{src/backend => plugins/kimchi}/iscsi.py | 0
{src/backend => plugins/kimchi}/isoinfo.py | 0
{src/backend => plugins/kimchi}/kvmusertests.py | 0
{src/backend => plugins/kimchi}/mockmodel.py | 0
{src/backend/control/vm => plugins/kimchi/model}/Makefile.am | 7 +-
{src/backend/control/vm => plugins/kimchi/model}/__init__.py | 8 ---
{src/backend => plugins/kimchi}/model/config.py | 0
{src/backend => plugins/kimchi}/model/cpuinfo.py | 0
{src/backend => plugins/kimchi}/model/debugreports.py | 0
{src/backend => plugins/kimchi}/model/diskutils.py | 0
{src/backend => plugins/kimchi}/model/featuretests.py | 0
{src/backend => plugins/kimchi}/model/groups.py | 0
{src/backend => plugins/kimchi}/model/host.py | 0
{src/backend => plugins/kimchi}/model/hostdev.py | 0
{src/backend => plugins/kimchi}/model/interfaces.py | 0
{src/backend => plugins/kimchi}/model/libvirtconnection.py | 0
{src/backend => plugins/kimchi}/model/libvirtstoragepool.py | 0
plugins/kimchi/model/model.py | 93
++++++++++++++++++++++++++++
{src/backend => plugins/kimchi}/model/networks.py | 0
{src/backend => plugins/kimchi}/model/peers.py | 0
{src/backend => plugins/kimchi}/model/storagepools.py | 0
{src/backend => plugins/kimchi}/model/storageservers.py | 0
{src/backend => plugins/kimchi}/model/storagetargets.py | 0
{src/backend => plugins/kimchi}/model/storagevolumes.py | 0
{src/backend => plugins/kimchi}/model/templates.py | 0
{src/backend => plugins/kimchi}/model/users.py | 0
{src/backend => plugins/kimchi}/model/utils.py | 0
{src/backend => plugins/kimchi}/model/vmhostdevs.py | 0
{src/backend => plugins/kimchi}/model/vmifaces.py | 0
{src/backend => plugins/kimchi}/model/vms.py | 0
{src/backend => plugins/kimchi}/model/vmsnapshots.py | 0
{src/backend => plugins/kimchi}/model/vmstorages.py | 0
{src/backend => plugins/kimchi}/osinfo.py | 0
{src/backend => plugins/kimchi}/scan.py | 0
{config => plugins/kimchi/ui}/Makefile.am | 0
{config/ui => plugins/kimchi/ui/config}/Makefile.am | 0
config/ui/tabs.xml => plugins/kimchi/ui/config/tab-ext.xml | 0
{src/backend => plugins/kimchi}/vmtemplate.py | 0
The remaining generic stuff on the "backend" side, usable or already in
use by other plugins, would be:
backend
|___model
| |___model.py
| |___plugins.py
| |___tasks.py
|___control
| |___base.py
| |___plugins.py
| |___tasks.py
| |___utils.py
|___xmlutils
| |___network.py
| |___disk.py
| |___utils.py
| |___graphics.py
| |___qemucmdline.py
| |___interface.py
|___asynctask.py
|___auth.py
|___basemodel.py
|___cachebust.py
|___distroloader.py
|___exception.py
|___i18n.py
|___netinfo.py
|___network.py
|___objectstore.py
|___proxy.py
|___repositories.py
|___rollbackcontext.py
|___root.py
|___screenshot.py
|___server.py
|___sslcert.py
|___swupdate.py
|___template.py
|___utils.py
|___vnc.py
We don't want to make this a too long e-mail and lose the focus on the
idea. We have further technical details to discuss in a follow up.
Please let us know your thoughts on that, any directions and
requirements you have on this, and how could we proceed with that plan.
--
Lucio Correia
Software Engineer
IBM LTC Brazil
9 years, 7 months
[kimchi-devel][PATCHv4 0/7] Upload storage volume
by lvroyce＠linux.vnet.ibm.com
From: Royce Lv <lvroyce(a)linux.vnet.ibm.com>
How to upload:
POST /storagepools/<pool-name>/storagevolumes/
{'capacity':size-of-the-volume,'name':name-of-the-volume,'format':'raw'}
PUT /storagepools/<pool-name>/storagevolumes/<volume-name>
{'chunk':chunk-data,'chunk_size':size-of-chunk,'index':index-of-the-chunk-of-uploaded-file}
PATCH 7/7 is a script against real model server,
you can use it to verify the logic of upload.
Royce Lv (7):
Update docs and json schema of storage volume upload
Update controller to make update accept formdata params
Add lock facility for storage volume upload
Update model for storage volume update
Fix incomplete record when uploading
update test case for storage volume upload
A test script against real model server
docs/API.md | 7 ++-
src/kimchi/API.json | 22 ++++++++
src/kimchi/control/base.py | 6 +--
src/kimchi/i18n.py | 4 ++
src/kimchi/isoinfo.py | 5 +-
src/kimchi/model/storagevolumes.py | 61 ++++++++++-----------
src/kimchi/model/utils.py | 46 ++++++++++++++++
tests/test_api.py | 108 +++++++++++++++++++++++++++++++++++++
tests/test_model_storagevolume.py | 95 ++++++++++++++++++++++++--------
9 files changed, 291 insertions(+), 63 deletions(-)
create mode 100644 tests/test_api.py
--
2.1.0
9 years, 7 months
[PATCH 0/5] Support vm name with special characters
by Ramon Medeiros
Ramon Medeiros (5):
Create title tag
Change the way that get_vm() and vm_list() search for VMs
Use get_vm() instead of conn.lookupByName
Encode vm_name to generate VNC token
Reflect the unicode conversion on token creation
src/kimchi/model/vms.py | 50 +++++++++++++++++++++++++++++++++---------
src/kimchi/model/vmstorages.py | 4 ++--
src/kimchi/vmtemplate.py | 4 +++-
ui/js/src/kimchi.api.js | 4 ++--
4 files changed, 47 insertions(+), 15 deletions(-)
--
2.1.0
9 years, 7 months
[PATCH v2] Supress error messages while checking vm metadata
by Ramon Medeiros
Changes
v2: Use parse functions present on model/utils.py
The function FeatureTests.disable_libvirt_error_logging only removes the
error message that will be displayed on the console. Libvirt also logs
errors on messages log. To avoid flood messages log with metadata
errors, it will be verified manually if the xml has the tag.
Signed-off-by: Ramon Medeiros <ramonn(a)linux.vnet.ibm.com>
---
src/kimchi/model/utils.py | 3 +++
1 file changed, 3 insertions(+)
diff --git a/src/kimchi/model/utils.py b/src/kimchi/model/utils.py
index b2739b2..1b114f6 100644
--- a/src/kimchi/model/utils.py
+++ b/src/kimchi/model/utils.py
@@ -95,6 +95,9 @@ def _kimchi_set_metadata_node(dom, node):
def libvirt_get_kimchi_metadata_node(dom, mode="current"):
+ if _kimchi_get_metadata_node(dom, "metadata") == None:
+ return
+
FeatureTests.disable_libvirt_error_logging()
try:
xml = dom.metadata(libvirt.VIR_DOMAIN_METADATA_ELEMENT,
--
2.1.0
9 years, 7 months
[PATCH 0/3] Snapshot enhancements
by Crístian Deives
Crístian Deives (3):
snapshot: Handle non-existing snapshots in mock lookup
snapshot: Change the action 'revert' to a Task
snapshot: Allow creating a snapshot of a domain in any state
src/kimchi/control/vm/snapshots.py | 2 +-
src/kimchi/i18n.py | 1 -
src/kimchi/mockmodel.py | 14 ++++++++++++++
src/kimchi/model/vmsnapshots.py | 36 ++++++++++++++++++++++--------------
tests/test_model.py | 15 +++++++++++----
tests/test_rest.py | 13 +++++++++++++
6 files changed, 61 insertions(+), 20 deletions(-)
--
2.1.0
9 years, 7 months
[PATCH] Do not resolve hostname to IP in using remote ISOs
by Aline Manera
Due a limitation in old versions of qemu-io (qemu-io was not able to resolve
hostnames preveting the ISO streaming support), Kimchi automatically
resolved the hostname to IP to workaround this problem.
As all the most used Linux distributions (RHEl7.1, Ubuntu 14.10, Fedora
21 and openSUSE 13.2) have a working version of qemu-io, Kimchi does not
need to care about it anymore.
It also fixes the issue #436
Signed-off-by: Aline Manera <alinefm(a)linux.vnet.ibm.com>
Signed-off-by: Daniel Henrique Barboza <danielhb(a)linux.vnet.ibm.com>
---
src/kimchi/model/config.py | 2 --
src/kimchi/model/featuretests.py | 26 --------------------------
src/kimchi/model/utils.py | 11 -----------
src/kimchi/model/vms.py | 1 -
src/kimchi/model/vmstorages.py | 10 +++-------
src/kimchi/vmtemplate.py | 12 ++----------
tests/test_model.py | 11 +----------
7 files changed, 6 insertions(+), 67 deletions(-)
diff --git a/src/kimchi/model/config.py b/src/kimchi/model/config.py
index a417c9b..ca0a74b 100644
--- a/src/kimchi/model/config.py
+++ b/src/kimchi/model/config.py
@@ -51,7 +51,6 @@ class CapabilitiesModel(object):
def __init__(self, **kargs):
self.conn = kargs['conn']
self.qemu_stream = False
- self.qemu_stream_dns = False
self.libvirt_stream_protocols = []
self.fc_host_support = False
self.metadata_support = False
@@ -88,7 +87,6 @@ class CapabilitiesModel(object):
kimchi_log.info("*** Running feature tests ***")
conn = self.conn.get()
self.qemu_stream = FeatureTests.qemu_supports_iso_stream()
- self.qemu_stream_dns = FeatureTests.qemu_iso_stream_dns()
self.nfs_target_probe = FeatureTests.libvirt_support_nfs_probe(conn)
self.fc_host_support = FeatureTests.libvirt_support_fc_host(conn)
self.metadata_support = FeatureTests.has_metadata_support(conn)
diff --git a/src/kimchi/model/featuretests.py b/src/kimchi/model/featuretests.py
index 8cbe31d..9400151 100644
--- a/src/kimchi/model/featuretests.py
+++ b/src/kimchi/model/featuretests.py
@@ -21,9 +21,7 @@ import cherrypy
import libvirt
import lxml.etree as ET
import platform
-import socket
import subprocess
-import threading
from lxml.builder import E
@@ -153,30 +151,6 @@ class FeatureTests(object):
return len(stderr) == 0
@staticmethod
- @servermethod
- def qemu_iso_stream_dns():
- host = socket.getfqdn(cherrypy.server.socket_host)
- port = cherrypy.server.socket_port
- cmd = ["qemu-io", "-r", "http://%s:%d/images/icon-fedora.png" %
- (host, port), "-c", "'read -v 0 512'"]
- proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,
- stderr=subprocess.PIPE)
-
- thread = threading.Thread(target=proc.communicate)
- thread.start()
- thread.join(5)
-
- if thread.is_alive():
- proc.kill()
- thread.join()
- return False
-
- if proc.returncode != 0:
- return False
-
- return True
-
- @staticmethod
def libvirt_support_fc_host(conn):
try:
FeatureTests.disable_libvirt_error_logging()
diff --git a/src/kimchi/model/utils.py b/src/kimchi/model/utils.py
index b2739b2..0c9d519 100644
--- a/src/kimchi/model/utils.py
+++ b/src/kimchi/model/utils.py
@@ -18,8 +18,6 @@
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
import libvirt
-import socket
-import urlparse
from lxml import etree, objectify
from lxml.builder import E, ElementMaker
@@ -41,15 +39,6 @@ def get_vm_name(vm_name, t_name, name_list):
raise OperationFailed("KCHUTILS0003E")
-def check_remote_disk_path(path, qemu_stream_dns):
- hostname = urlparse.urlparse(path).hostname
- if hostname is not None and not qemu_stream_dns:
- ip = socket.gethostbyname(hostname)
- return path.replace(hostname, ip)
-
- return path
-
-
def get_vm_config_flag(dom, mode="persistent"):
# libvirt.VIR_DOMAIN_AFFECT_CURRENT is 0
# VIR_DOMAIN_AFFECT_LIVE is 1, VIR_DOMAIN_AFFECT_CONFIG is 2
diff --git a/src/kimchi/model/vms.py b/src/kimchi/model/vms.py
index c8267c1..a6ca27b 100644
--- a/src/kimchi/model/vms.py
+++ b/src/kimchi/model/vms.py
@@ -125,7 +125,6 @@ class VMsModel(object):
stream_protocols = self.caps.libvirt_stream_protocols
xml = t.to_vm_xml(name, vm_uuid,
libvirt_stream_protocols=stream_protocols,
- qemu_stream_dns=self.caps.qemu_stream_dns,
graphics=graphics,
volumes=vol_list)
diff --git a/src/kimchi/model/vmstorages.py b/src/kimchi/model/vmstorages.py
index 37aca64..87c6b3d 100644
--- a/src/kimchi/model/vmstorages.py
+++ b/src/kimchi/model/vmstorages.py
@@ -1,7 +1,7 @@
#
# Project Kimchi
#
-# Copyright IBM, Corp. 2014
+# Copyright IBM, Corp. 2014-2015
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
@@ -26,7 +26,7 @@ from kimchi.exception import OperationFailed
from kimchi.model.config import CapabilitiesModel
from kimchi.model.vms import DOM_STATE_MAP, VMModel
from kimchi.model.storagevolumes import StorageVolumeModel
-from kimchi.model.utils import check_remote_disk_path, get_vm_config_flag
+from kimchi.model.utils import get_vm_config_flag
from kimchi.osinfo import lookup
from kimchi.model.diskutils import get_disk_ref_cnt, set_disk_ref_cnt
from kimchi.utils import kimchi_log
@@ -131,8 +131,6 @@ class VMStoragesModel(object):
params['disk'] = vol_info['type']
params.update(self._get_available_bus_address(params['bus'], vm_name))
- params['path'] = check_remote_disk_path(params['path'],
- self.caps.qemu_stream_dns)
# Add device to VM
dev, xml = get_disk_xml(params)
@@ -215,9 +213,7 @@ class VMStorageModel(object):
if dev_info['type'] != 'cdrom':
raise InvalidOperation("KCHVMSTOR0006E")
- params['path'] = check_remote_disk_path(params.get('path', ''),
- self.caps.qemu_stream_dns)
-
+ params['path'] = params.get('path', '')
old_disk_path = dev_info['path']
new_disk_path = params['path']
if new_disk_path != old_disk_path:
diff --git a/src/kimchi/vmtemplate.py b/src/kimchi/vmtemplate.py
index ef97d83..e047228 100644
--- a/src/kimchi/vmtemplate.py
+++ b/src/kimchi/vmtemplate.py
@@ -18,7 +18,6 @@
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
import os
-import socket
import stat
import time
import urlparse
@@ -128,7 +127,7 @@ class VMTemplate(object):
except IsoFormatError:
raise InvalidParameter("KCHISO0001E", {'filename': iso})
- def _get_cdrom_xml(self, libvirt_stream_protocols, qemu_stream_dns):
+ def _get_cdrom_xml(self, libvirt_stream_protocols):
if 'cdrom' not in self.info:
return ''
@@ -139,11 +138,6 @@ class VMTemplate(object):
params['index'] = self.info['cdrom_index']
params['path'] = self.info['cdrom']
- hostname = urlparse.urlparse(params['path']).hostname
- if hostname is not None and not qemu_stream_dns:
- ip = socket.gethostbyname(hostname)
- params['path'] = params['path'].replace(hostname, ip)
-
if self.info.get('iso_stream', False):
protocol = urlparse.urlparse(params['path']).scheme
if protocol not in libvirt_stream_protocols:
@@ -304,10 +298,8 @@ class VMTemplate(object):
graphics.update(kwargs.get('graphics', {}))
params['graphics'] = get_graphics_xml(graphics)
- qemu_stream_dns = kwargs.get('qemu_stream_dns', False)
libvirt_stream_protocols = kwargs.get('libvirt_stream_protocols', [])
- cdrom_xml = self._get_cdrom_xml(libvirt_stream_protocols,
- qemu_stream_dns)
+ cdrom_xml = self._get_cdrom_xml(libvirt_stream_protocols)
if not urlparse.urlparse(self.info.get('cdrom', "")).scheme in \
libvirt_stream_protocols and \
diff --git a/tests/test_model.py b/tests/test_model.py
index 210adfd..bd195b5 100644
--- a/tests/test_model.py
+++ b/tests/test_model.py
@@ -23,9 +23,7 @@ import os
import pwd
import re
import shutil
-import socket
import time
-import urlparse
import unittest
import uuid
@@ -564,14 +562,7 @@ class ModelTests(unittest.TestCase):
{'path': valid_remote_iso_path})
cdrom_info = inst.vmstorage_lookup(vm_name, cdrom_dev)
cur_cdrom_path = re.sub(":80/", '/', cdrom_info['path'])
-
- # As Kimchi server is not running during this test case
- # CapabilitiesModel.qemu_stream_dns will be always False
- # so we need to convert the hostname to IP
- output = urlparse.urlparse(valid_remote_iso_path)
- hostname = socket.gethostbyname(output.hostname)
- url = valid_remote_iso_path.replace(output.hostname, hostname)
- self.assertEquals(url, cur_cdrom_path)
+ self.assertEquals(valid_remote_iso_path, cur_cdrom_path)
@unittest.skipUnless(utils.running_as_root(), 'Must be run as root')
def test_vm_storage_provisioning(self):
--
2.1.0
9 years, 7 months
[PATCH 0/7] Make Template defaults configurable
by Aline Manera
Aline Manera (7):
Remove useless variable in osinfo.py
Merge common_spec wiht defaults value in osinfo.py
Make Template defaults configurable
Add libvirt-daemon-config-network package as Kimchi dependency
Verify all networks set as Template defaults prior to server start up
Create option to auto create ISO pool or not on server start up
Verify storage pool set as Template default prior to server starts up
contrib/DEBIAN/control.in | 1 +
contrib/kimchi.spec.fedora.in | 3 ++
contrib/kimchi.spec.suse.in | 3 ++
docs/README.md | 14 +++---
src/Makefile.am | 2 +-
src/kimchi.conf.in | 3 ++
src/kimchi/config.py.in | 1 +
src/kimchi/model/model.py | 44 +----------------
src/kimchi/model/networks.py | 61 ++++++++++-------------
src/kimchi/model/storagepools.py | 66 +++++++++++++++++++++++++
src/kimchi/model/templates.py | 2 -
src/kimchi/osinfo.py | 101 ++++++++++++++++++++++++++++++---------
src/kimchid.in | 3 ++
src/template.conf | 47 ++++++++++++++++++
14 files changed, 240 insertions(+), 111 deletions(-)
create mode 100644 src/template.conf
--
2.1.0
9 years, 8 months
[PATCH 0/6 v7] Asyn VM creation
by Aline Manera
V6 -> V7:
- Update test cases
- Update UI to reuse the clone box for pending VM creation
Aline Manera (1):
Create VMs Asynchronously: UI
Christy Perez (5):
Append clone to target_uri for vm clone task
Tests for new clone target_uri
UI changes for new clone target_uri
Create VMs asynchronously: Backend
Create VMs Asynchronously: Tests
src/kimchi/control/vms.py | 4 +--
src/kimchi/model/vms.py | 33 ++++++++++++++++++-----
tests/test_authorization.py | 25 +++++++++++-------
tests/test_mockmodel.py | 14 +++++++---
tests/test_model.py | 49 +++++++++++++++++++++++-----------
tests/test_rest.py | 60 +++++++++++++++++++++++++++++++-----------
tests/test_server.py | 26 ++++++++----------
ui/css/theme-default/list.css | 8 +++---
ui/js/src/kimchi.guest_main.js | 32 ++++++++++++++++++----
ui/pages/guest.html.tmpl | 4 +--
ui/pages/i18n.json.tmpl | 1 +
11 files changed, 176 insertions(+), 80 deletions(-)
--
2.1.0
9 years, 8 months
