[PATCH V2 0/4] Implement integrity verification: verify template integrity
From: ShaoHe Feng <shaohef@linux.vnet.ibm.com> V1 -> V2 rename invalid_integrity to validate_integrity change list to array in API.md list is the python semantic, array is json semantic. Implement integrity verification: verify template integrity, update API.md Sometimes, user create a template, but networks, cdrom, disks or storagepool will change later. So users can not create a vm from this template successfully. It is necessary to check some paramenters of template. ShaoHe Feng (4): Implement integrity verification: verify template integrity, update API.md add a new method to get iso info for VMTemplate class Implement integrity verification: verify template integrity in backend Implement integrity verification: update test case docs/API.md | 4 +++ src/kimchi/control/templates.py | 1 + src/kimchi/mockmodel.py | 5 +++- src/kimchi/model/templates.py | 7 +++++- src/kimchi/vmtemplate.py | 54 ++++++++++++++++++++++++++++++++--------- tests/test_model.py | 33 +++++++++++++++++++++++++ tests/test_rest.py | 40 ++++++++++++++++++++++++++++++ 7 files changed, 131 insertions(+), 13 deletions(-) -- 1.8.4.2
From: ShaoHe Feng <shaohef@linux.vnet.ibm.com> Sometimes, user create a template, but networks, cdrom, disks or storagepool will change later. So users can not create a vm from this template successfully. It is necessary to check some paramenters of template. This patch will check the follow paramenters of template. networks: check networks exists. cdrom: check cdrom is available. disks: check the volume is available. This patch does not check the storagepool exists. waiting for royce's disks patch. Signed-off-by: ShaoHe Feng <shaohef@linux.vnet.ibm.com> --- docs/API.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/API.md b/docs/API.md index fff740d..a7d39e9 100644 --- a/docs/API.md +++ b/docs/API.md @@ -249,6 +249,10 @@ A interface represents available network interface on VM. Independent Computing Environments * null: Graphics is disabled or type not supported * listen: The network which the vnc/spice server listens on. + * invalid: A dict indicates which paramenters of this template are invalid. + * networks *(optional)*: An array of invalid network names. + * cdrom *(optional)*: An array of invalid cdrom names. + * disks *(optional)*: An array of invalid volume names. * **DELETE**: Remove the Template * **POST**: *See Template Actions* -- 1.8.4.2
From: ShaoHe Feng <shaohef@linux.vnet.ibm.com> move the related code about get iso info to a new function. Then template integrity can make use of it. Signed-off-by: ShaoHe Feng <shaohef@linux.vnet.ibm.com> --- src/kimchi/vmtemplate.py | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/src/kimchi/vmtemplate.py b/src/kimchi/vmtemplate.py index 5767a13..a11ddd0 100644 --- a/src/kimchi/vmtemplate.py +++ b/src/kimchi/vmtemplate.py @@ -56,20 +56,10 @@ class VMTemplate(object): iso = args.get('cdrom', '') if scan and len(iso) > 0: - - iso_prefixes = ['/', 'http', 'https', 'ftp', 'ftps', 'tftp'] - if len(filter(iso.startswith, iso_prefixes)) == 0: - raise InvalidParameter("KCHTMPL0006E", {'param': iso}) - + iso_distro, iso_version = self.get_iso_info(iso) if not iso.startswith('/'): self.info.update({'iso_stream': True}) - try: - iso_img = IsoImage(iso) - iso_distro, iso_version = iso_img.probe() - except IsoFormatError: - raise InvalidParameter("KCHISO0001E", {'filename': iso}) - # Fetch defaults based on the os distro and version os_distro = args.get('os_distro', iso_distro) os_version = args.get('os_version', iso_version) @@ -84,6 +74,16 @@ class VMTemplate(object): args['graphics'] = graphics self.info.update(args) + def get_iso_info(self, iso): + iso_prefixes = ['/', 'http', 'https', 'ftp', 'ftps', 'tftp'] + if len(filter(iso.startswith, iso_prefixes)) == 0: + raise InvalidParameter("KCHTMPL0006E", {'param': iso}) + try: + iso_img = IsoImage(iso) + return iso_img.probe() + except IsoFormatError: + raise InvalidParameter("KCHISO0001E", {'filename': iso}) + def _get_cdrom_xml(self, libvirt_stream, qemu_stream_dns): bus = self.info['cdrom_bus'] dev = "%s%s" % (self._bus_to_dev[bus], -- 1.8.4.2
From: ShaoHe Feng <shaohef@linux.vnet.ibm.com> update controller, mockmodel and model Signed-off-by: ShaoHe Feng <shaohef@linux.vnet.ibm.com> --- src/kimchi/control/templates.py | 1 + src/kimchi/mockmodel.py | 5 ++++- src/kimchi/model/templates.py | 7 ++++++- src/kimchi/vmtemplate.py | 32 ++++++++++++++++++++++++++++++++ 4 files changed, 43 insertions(+), 2 deletions(-) diff --git a/src/kimchi/control/templates.py b/src/kimchi/control/templates.py index 8135e32..3f1dc7a 100644 --- a/src/kimchi/control/templates.py +++ b/src/kimchi/control/templates.py @@ -46,6 +46,7 @@ class Template(Resource): def data(self): return {'name': self.ident, 'icon': self.info['icon'], + 'invalid': self.info['invalid'], 'os_distro': self.info['os_distro'], 'os_version': self.info['os_version'], 'cpus': self.info['cpus'], diff --git a/src/kimchi/mockmodel.py b/src/kimchi/mockmodel.py index a0e5120..bc244fa 100644 --- a/src/kimchi/mockmodel.py +++ b/src/kimchi/mockmodel.py @@ -207,7 +207,7 @@ class MockModel(object): def template_lookup(self, name): t = self._get_template(name) - return t.info + return t.validate_integrity() def template_delete(self, name): try: @@ -809,6 +809,9 @@ class MockVMTemplate(VMTemplate): VMTemplate.__init__(self, args) self.model = mockmodel_inst + def _get_all_networks_name(self): + return self.model.networks_get_list() + def _storage_validate(self): pool_uri = self.info['storagepool'] pool_name = pool_name_from_uri(pool_uri) diff --git a/src/kimchi/model/templates.py b/src/kimchi/model/templates.py index 5d09813..87ea21e 100644 --- a/src/kimchi/model/templates.py +++ b/src/kimchi/model/templates.py @@ -21,6 +21,7 @@ # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA import copy +import os import libvirt @@ -83,7 +84,7 @@ class TemplateModel(object): def lookup(self, name): t = self.get_template(name, self.objstore, self.conn) - return t.info + return t.validate_integrity() def clone(self, name): # set default name @@ -158,6 +159,10 @@ class LibvirtVMTemplate(VMTemplate): return pool + def _get_all_networks_name(self): + conn = self.conn.get() + return [net.name() for net in conn.listAllNetworks()] + def _network_validate(self): names = self.info['networks'] for name in names: diff --git a/src/kimchi/vmtemplate.py b/src/kimchi/vmtemplate.py index a11ddd0..af07ee3 100644 --- a/src/kimchi/vmtemplate.py +++ b/src/kimchi/vmtemplate.py @@ -344,3 +344,35 @@ class VMTemplate(object): def _get_storage_type(self): return '' + + def _get_all_networks_name(self): + return [] + + def validate_integrity(self): + invalid = {} + # validate networks integrity + invalid_networks = list(set(self.info['networks']) - + set(self._get_all_networks_name())) + if invalid_networks: + invalid['networks'] = invalid_networks + + # validate iso integrity + # FIXME when we support multiples cdrom devices + iso = self.info['cdrom'] + try: + self.get_iso_info(iso) + except Exception: + invalid['cdrom'] = [iso] + + # validate disks integrity + volumes = [] + for disk in self.info['disks']: + volume = disk.get("volume") + if volume is not None and not os.path.exists(volume): + volumes.append(volume) + if volumes: + invalid['disks'] = volumes + + self.info['invalid'] = invalid + + return self.info -- 1.8.4.2
From: ShaoHe Feng <shaohef@linux.vnet.ibm.com> update test_model.py and test_rest.py Signed-off-by: ShaoHe Feng <shaohef@linux.vnet.ibm.com> --- tests/test_model.py | 33 +++++++++++++++++++++++++++++++++ tests/test_rest.py | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 73 insertions(+) diff --git a/tests/test_model.py b/tests/test_model.py index 73d89bf..f7ab420 100644 --- a/tests/test_model.py +++ b/tests/test_model.py @@ -24,6 +24,7 @@ import os import platform import psutil +import shutil import tempfile import threading import time @@ -441,6 +442,38 @@ class ModelTests(unittest.TestCase): self.assertEquals(params[key], info[key]) @unittest.skipUnless(utils.running_as_root(), 'Must be run as root') + def test_template_integrity(self): + inst = model.Model('test:///default', + objstore_loc=self.tmp_store) + + with RollbackContext() as rollback: + net_name = 'test-network' + net_args = {'name': net_name, + 'connection': 'nat', + 'subnet': '127.0.100.0/24'} + inst.networks_create(net_args) + + path = '/tmp/kimchi-iso/' + if not os.path.exists(path): + os.makedirs(path) + iso = path + 'ubuntu12.04.iso' + iso_gen.construct_fake_iso(iso, True, '12.04', 'ubuntu') + + params = {'name': 'test', 'memory': 1024, 'cpus': 1, + 'networks': ['test-network'], 'cdrom': iso, + 'disks': [{'volume':iso}]} + inst.templates_create(params) + rollback.prependDefer(inst.template_delete, 'test') + + inst.network_delete(net_name) + shutil.rmtree(path) + + info = inst.template_lookup('test') + self.assertEquals(info['invalid']['cdrom'], [iso]) + self.assertEquals(info['invalid']['networks'], [net_name]) + self.assertEquals(info['invalid']['disks'], [iso]) + + @unittest.skipUnless(utils.running_as_root(), 'Must be run as root') def test_template_clone(self): inst = model.Model('qemu:///system', objstore_loc=self.tmp_store) diff --git a/tests/test_rest.py b/tests/test_rest.py index 26078d6..0b3ec76 100644 --- a/tests/test_rest.py +++ b/tests/test_rest.py @@ -25,6 +25,7 @@ import base64 import json import os import random +import shutil import time import unittest @@ -32,6 +33,7 @@ import unittest from functools import partial +import iso_gen import kimchi.mockmodel import kimchi.server from kimchi.rollbackcontext import RollbackContext @@ -1073,6 +1075,44 @@ class RestTests(unittest.TestCase): resp = self.request('/templates/%s' % tmpl_name, '{}', 'DELETE') self.assertEquals(204, resp.status) + def test_template_integrity(self): + + path = '/tmp/kimchi-iso/' + if not os.path.exists(path): + os.makedirs(path) + iso = path + 'ubuntu12.04.iso' + iso_gen.construct_fake_iso(iso, True, '12.04', 'ubuntu') + + req = json.dumps({'name': 'test-network', + 'connection': 'nat', + 'net': '127.0.1.0/24'}) + resp = request(host, port, '/networks', req, 'POST') + self.assertEquals(201, resp.status) + + + t = {'name': 'test', 'memory': 1024, 'cpus': 1, + 'networks': ['test-network'], 'cdrom': iso, + 'disks': [{'volume':iso}]} + + req = json.dumps(t) + resp = self.request('/templates', req, 'POST') + self.assertEquals(201, resp.status) + + shutil.rmtree(path) + # Delete the network + resp = request(host, port, '/networks/test-network', '{}', 'DELETE') + self.assertEquals(204, resp.status) + + # Verify the template + res = json.loads(self.request('/templates/test').read()) + self.assertEquals(res['invalid']['cdrom'], [iso]) + self.assertEquals(res['invalid']['networks'], ['test-network']) + self.assertEquals(res['invalid']['disks'], [iso]) + + # Delete the template + resp = request(host, port, '/templates/test', '{}', 'DELETE') + self.assertEquals(204, resp.status) + def test_iso_scan_shallow(self): # fake environment preparation self._create_pool('pool-3') -- 1.8.4.2
I got the following errors while running the tests Seems listAllNetworks() is not supported in all libvirt versions. ====================================================================== ERROR: test_template_clone (test_model.ModelTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/alinefm/kimchi/tests/test_model.py", line 483, in test_template_clone orig_temp = inst.template_lookup(orig_params['name']) File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 87, in lookup return t.validate_integrity() File "/home/alinefm/kimchi/src/kimchi/vmtemplate.py", line 355, in validate_integrity set(self._get_all_networks_name())) File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 164, in _get_all_networks_name return [net.name() for net in conn.listAllNetworks()] AttributeError: virConnect instance has no attribute 'listAllNetworks' ====================================================================== ERROR: test_template_create (test_model.ModelTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/alinefm/kimchi/tests/test_model.py", line 427, in test_template_create info = inst.template_lookup('test') File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 87, in lookup return t.validate_integrity() File "/home/alinefm/kimchi/src/kimchi/vmtemplate.py", line 355, in validate_integrity set(self._get_all_networks_name())) File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 164, in _get_all_networks_name return [net.name() for net in conn.listAllNetworks()] AttributeError: virConnect instance has no attribute 'listAllNetworks' ====================================================================== ERROR: test_template_integrity (test_model.ModelTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/alinefm/kimchi/tests/test_model.py", line 471, in test_template_integrity info = inst.template_lookup('test') File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 87, in lookup return t.validate_integrity() File "/home/alinefm/kimchi/src/kimchi/vmtemplate.py", line 355, in validate_integrity set(self._get_all_networks_name())) File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 164, in _get_all_networks_name return [net.name() for net in conn.listAllNetworks()] AttributeError: virConnect instance has no attribute 'listAllNetworks' ====================================================================== ERROR: test_template_storage_customise (test_model.ModelTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/alinefm/kimchi/tests/test_model.py", line 381, in test_template_storage_customise 'test', params) File "/usr/lib/python2.7/unittest/case.py", line 476, in assertRaises callableObj(*args, **kwargs) File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 110, in update old_t = self.lookup(name) File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 87, in lookup return t.validate_integrity() File "/home/alinefm/kimchi/src/kimchi/vmtemplate.py", line 355, in validate_integrity set(self._get_all_networks_name())) File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 164, in _get_all_networks_name return [net.name() for net in conn.listAllNetworks()] AttributeError: virConnect instance has no attribute 'listAllNetworks' ====================================================================== ERROR: test_template_update (test_model.ModelTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/alinefm/kimchi/tests/test_model.py", line 508, in test_template_update self.assertEquals('new-test', inst.template_update('test', params)) File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 110, in update old_t = self.lookup(name) File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 87, in lookup return t.validate_integrity() File "/home/alinefm/kimchi/src/kimchi/vmtemplate.py", line 355, in validate_integrity set(self._get_all_networks_name())) File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 164, in _get_all_networks_name return [net.name() for net in conn.listAllNetworks()] AttributeError: virConnect instance has no attribute 'listAllNetworks' ---------------------------------------------------------------------- Ran 157 tests in 128.049s FAILED (errors=5) [20/Feb/2014:21:50:29] ENGINE Waiting for child threads to terminate... make[3]: *** [check-local] Error 1 make[3]: Leaving directory `/home/alinefm/kimchi/tests' make[2]: *** [check-am] Error 2 make[2]: Leaving directory `/home/alinefm/kimchi/tests' make[1]: *** [check] Error 2 make[1]: Leaving directory `/home/alinefm/kimchi/tests' make: *** [check-recursive] Error 1 On 02/20/2014 03:00 AM, shaohef@linux.vnet.ibm.com wrote:
From: ShaoHe Feng <shaohef@linux.vnet.ibm.com>
V1 -> V2 rename invalid_integrity to validate_integrity change list to array in API.md list is the python semantic, array is json semantic.
Implement integrity verification: verify template integrity, update API.md
Sometimes, user create a template, but networks, cdrom, disks or storagepool will change later. So users can not create a vm from this template successfully. It is necessary to check some paramenters of template.
ShaoHe Feng (4): Implement integrity verification: verify template integrity, update API.md add a new method to get iso info for VMTemplate class Implement integrity verification: verify template integrity in backend Implement integrity verification: update test case
docs/API.md | 4 +++ src/kimchi/control/templates.py | 1 + src/kimchi/mockmodel.py | 5 +++- src/kimchi/model/templates.py | 7 +++++- src/kimchi/vmtemplate.py | 54 ++++++++++++++++++++++++++++++++--------- tests/test_model.py | 33 +++++++++++++++++++++++++ tests/test_rest.py | 40 ++++++++++++++++++++++++++++++ 7 files changed, 131 insertions(+), 13 deletions(-)
On 02/21/2014 08:52 AM, Aline Manera wrote:
I got the following errors while running the tests Seems listAllNetworks() is not supported in all libvirt versions.
sure, should use conn.listNetworks() + conn.listDefinedNetworks()
====================================================================== ERROR: test_template_clone (test_model.ModelTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/alinefm/kimchi/tests/test_model.py", line 483, in test_template_clone orig_temp = inst.template_lookup(orig_params['name']) File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 87, in lookup return t.validate_integrity() File "/home/alinefm/kimchi/src/kimchi/vmtemplate.py", line 355, in validate_integrity set(self._get_all_networks_name())) File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 164, in _get_all_networks_name return [net.name() for net in conn.listAllNetworks()] AttributeError: virConnect instance has no attribute 'listAllNetworks'
====================================================================== ERROR: test_template_create (test_model.ModelTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/alinefm/kimchi/tests/test_model.py", line 427, in test_template_create info = inst.template_lookup('test') File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 87, in lookup return t.validate_integrity() File "/home/alinefm/kimchi/src/kimchi/vmtemplate.py", line 355, in validate_integrity set(self._get_all_networks_name())) File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 164, in _get_all_networks_name return [net.name() for net in conn.listAllNetworks()] AttributeError: virConnect instance has no attribute 'listAllNetworks'
====================================================================== ERROR: test_template_integrity (test_model.ModelTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/alinefm/kimchi/tests/test_model.py", line 471, in test_template_integrity info = inst.template_lookup('test') File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 87, in lookup return t.validate_integrity() File "/home/alinefm/kimchi/src/kimchi/vmtemplate.py", line 355, in validate_integrity set(self._get_all_networks_name())) File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 164, in _get_all_networks_name return [net.name() for net in conn.listAllNetworks()] AttributeError: virConnect instance has no attribute 'listAllNetworks'
====================================================================== ERROR: test_template_storage_customise (test_model.ModelTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/alinefm/kimchi/tests/test_model.py", line 381, in test_template_storage_customise 'test', params) File "/usr/lib/python2.7/unittest/case.py", line 476, in assertRaises callableObj(*args, **kwargs) File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 110, in update old_t = self.lookup(name) File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 87, in lookup return t.validate_integrity() File "/home/alinefm/kimchi/src/kimchi/vmtemplate.py", line 355, in validate_integrity set(self._get_all_networks_name())) File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 164, in _get_all_networks_name return [net.name() for net in conn.listAllNetworks()] AttributeError: virConnect instance has no attribute 'listAllNetworks'
====================================================================== ERROR: test_template_update (test_model.ModelTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/alinefm/kimchi/tests/test_model.py", line 508, in test_template_update self.assertEquals('new-test', inst.template_update('test', params)) File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 110, in update old_t = self.lookup(name) File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 87, in lookup return t.validate_integrity() File "/home/alinefm/kimchi/src/kimchi/vmtemplate.py", line 355, in validate_integrity set(self._get_all_networks_name())) File "/home/alinefm/kimchi/src/kimchi/model/templates.py", line 164, in _get_all_networks_name return [net.name() for net in conn.listAllNetworks()] AttributeError: virConnect instance has no attribute 'listAllNetworks'
---------------------------------------------------------------------- Ran 157 tests in 128.049s
FAILED (errors=5) [20/Feb/2014:21:50:29] ENGINE Waiting for child threads to terminate... make[3]: *** [check-local] Error 1 make[3]: Leaving directory `/home/alinefm/kimchi/tests' make[2]: *** [check-am] Error 2 make[2]: Leaving directory `/home/alinefm/kimchi/tests' make[1]: *** [check] Error 2 make[1]: Leaving directory `/home/alinefm/kimchi/tests' make: *** [check-recursive] Error 1
On 02/20/2014 03:00 AM, shaohef@linux.vnet.ibm.com wrote:
From: ShaoHe Feng <shaohef@linux.vnet.ibm.com>
V1 -> V2 rename invalid_integrity to validate_integrity change list to array in API.md list is the python semantic, array is json semantic.
Implement integrity verification: verify template integrity, update API.md Sometimes, user create a template, but networks, cdrom, disks or storagepool will change later. So users can not create a vm from this template successfully. It is necessary to check some paramenters of template.
ShaoHe Feng (4): Implement integrity verification: verify template integrity, update API.md add a new method to get iso info for VMTemplate class Implement integrity verification: verify template integrity in backend Implement integrity verification: update test case
docs/API.md | 4 +++ src/kimchi/control/templates.py | 1 + src/kimchi/mockmodel.py | 5 +++- src/kimchi/model/templates.py | 7 +++++- src/kimchi/vmtemplate.py | 54 ++++++++++++++++++++++++++++++++--------- tests/test_model.py | 33 +++++++++++++++++++++++++ tests/test_rest.py | 40 ++++++++++++++++++++++++++++++ 7 files changed, 131 insertions(+), 13 deletions(-)
-- Thanks and best regards! Sheldon Feng(冯少合)<shaohef@linux.vnet.ibm.com> IBM Linux Technology Center
participants (3)
-
Aline Manera -
shaohef＠linux.vnet.ibm.com -
Sheldon