July 2014 - Kimchi-devel - oVirt List Archives

[PATCH v8 0/4] Host device passthrough: Summary
by Zhou Zheng Sheng 29 Jul '14

29 Jul '14

Hi, This patch series is to enable Kimchi to assign hos devices directly to a VM, thus greately improve VM performance. Currently we support assigning PCI device, USB device and SCSI LUN. For example, we can assign an NIC to VM to improve guest network throughput, or passthrough a USB camera to enable the guest OS to record video. Host devices form a tree. We can assign most of the devices in the tree to VM. By assigning a device, all the devices in its sub-tree are also assigned. It might not make sense to assign a USB controller, because the host may be using one of the devices connected to the controller. Instead, Kimchi just presents the "leaf" devices to assign to guest. In recent Linux kernel and KVM, it is able to recognize the IOMMU group of a PCI device. The "leaf" PCI devices in the same IOMMU group should be assigned and dismissed together. The IOMMU group is the actual smallest isolation granularity of the PCI devices. The first patch is to list all host devices information. It's useful on its own to show host devices information. The second patch is to list all eligible host devices to assign, as well as the "affected" devices in the same IOMMU group. The third patch creates a sub-collection "hostdevs" to the VM resource, and deals with assigning and dismissing devices. The fourth patch adds a sub-collection "vm_holders" to the host device resource. It's to list all VMs that are holding the device. I'll update API and unit test once everyone is happy with the interface and logic. v8: Change the URI for listing affected passthrough devices, suggested by Aline. Move hostdev.py to src/kimchi/model.py. Still discussing API with the front-end developer. Once the API accepted, The author should add tests for model and mockmodel, and change API.md and API.json accordingly. v7: Some minor coding style improvements. v6: Do not passthrough PCI device of class code 0x07. It might contains system device not suitable to assign to guest. v5: Filter ealigible pci devices according to pci class. When assigning a device to VM, check if there are other VMs holding it. Use "kimchi.model.utils.get_vm_config_flag()" to correctly set the device attaching API flag. v4: Add new sub-collection to host device to list the VMs holding the device. v3: Fix a small naming error introduced by rebase. v2: Handle the devices in VM's sub-collection "hostdevs". v1: Handle the devices in VM template. Zhou Zheng Sheng (4): Host device passthrough: List all types of host devices Host device passthrough: List eligible device to passthrough Host device passthrough: Directly assign and dissmis host device from VM Host device passthrough: List VMs that are holding a host device docs/API.md | 11 +- src/kimchi/control/host.py | 7 + src/kimchi/control/vm/hostdevs.py | 44 +++++ src/kimchi/featuretests.py | 10 +- src/kimchi/i18n.py | 10 + src/kimchi/mockmodel.py | 7 +- src/kimchi/model/config.py | 2 + src/kimchi/model/host.py | 38 ++-- src/kimchi/model/hostdev.py | 311 +++++++++++++++++++++++++++++++ src/kimchi/model/libvirtstoragepool.py | 18 +- src/kimchi/model/vmhostdevs.py | 324 +++++++++++++++++++++++++++++++++ src/kimchi/rollbackcontext.py | 3 + src/kimchi/xmlutils.py | 26 ++- tests/test_rest.py | 6 +- tests/test_storagepool.py | 7 +- 15 files changed, 784 insertions(+), 40 deletions(-) create mode 100644 src/kimchi/control/vm/hostdevs.py create mode 100644 src/kimchi/model/hostdev.py create mode 100644 src/kimchi/model/vmhostdevs.py -- 1.9.3

3 19

Re: [Kimchi-devel] [PATCH] bug fix: Avoid equals sign in VM console URL
by Sheldon 29 Jul '14

29 Jul '14

On 07/29/2014 10:22 AM, Zheng Sheng ZS Zhou wrote: > > Sheldon <shaohef(a)linux.vnet.ibm.com> wrote 2014-07-28 17:23:21: > > > From: Sheldon <shaohef(a)linux.vnet.ibm.com> > > To: alinefm(a)linux.vnet.ibm.com, Kimchi Devel <kimchi- > > devel(a)ovirt.org>, Zheng Sheng ZS Zhou/China/IBM@IBMCN > > Date: 2014-07-28 17:22 > > Subject: Re: [Kimchi-devel] [PATCH] bug fix: Avoid equals sign in VM > > console URL > > > > On 07/26/2014 05:01 AM, alinefm(a)linux.vnet.ibm.com wrote: > > > From: Aline Manera <alinefm(a)linux.vnet.ibm.com> > > > > > > >From python documentation, base64.urlsafe_b64encode(s) > > substitutes - instead > > > of + and _ instead of / in the standard Base64 alphabet, BUT the > result can > > > still contain = which is not safe in a URL query component. > > > As token value is not decoded nowhere, replace = by A > > also in our kimchi I have try: > > In [45]: base64.urlsafe_b64encode("abcd") > > Out[45]: 'YWJjZA==' > > In [41]: base64.urlsafe_b64encode("abcde") > > Out[41]: 'YWJjZGU=' > > > > > > JS is very cool, it can decode base64 without "=" padding well > > kimchi.urlSafeB64Decode("YWJjZA") > > "abcd" > > kimchi.urlSafeB64Decode("YWJjZGU") > > "abcde" > > > > we just need in python: > > In [48]: base64.urlsafe_b64encode("abcd").rstrip("=") > > Out[48]: 'YWJjZA' > > > > > > A friendly remind: We have to make sure this trick also works in IE > and Chome. > Try chrom can work. But No IE environment. > > > > > The problem with equals sign was only identified on Spice connections. > > > noVNC can deal well with that. > > > > > > For reference: https://docs.python.org/2/library/base64.html > > > > > > Signed-off-by: Aline Manera <alinefm(a)linux.vnet.ibm.com> > > > --- > > > src/kimchi/vnc.py | 9 ++++++++- > > > ui/js/src/kimchi.api.js | 18 ++++++++++++++++-- > > > 2 files changed, 24 insertions(+), 3 deletions(-) > > > > > > diff --git a/src/kimchi/vnc.py b/src/kimchi/vnc.py > > > index 9380e21..4159049 100644 > > > --- a/src/kimchi/vnc.py > > > +++ b/src/kimchi/vnc.py > > > @@ -54,7 +54,14 @@ def new_ws_proxy(): > > > > > > def add_proxy_token(name, port): > > > with open(os.path.join(WS_TOKENS_DIR, name), 'w') as f: > > > - name = base64.urlsafe_b64encode(name) > > > + """ > > > + From python documentation base64.urlsafe_b64encode(s) > > > + substitutes - instead of + and _ instead of / in the > > > + standard Base64 alphabet, BUT the result can still > > > + contain = which is not safe in a URL query component. > > > + As token value is not decoded nowhere, replace = by A > > > + """ > > > + name = base64.urlsafe_b64encode(name).replace('=', 'A') > > > f.write('%s: localhost:%s' % (name.encode('utf-8'), port)) > > > > > > > > > diff --git a/ui/js/src/kimchi.api.js b/ui/js/src/kimchi.api.js > > > index 8f5b68f..30360c5 100644 > > > --- a/ui/js/src/kimchi.api.js > > > +++ b/ui/js/src/kimchi.api.js > > > @@ -352,7 +352,14 @@ var kimchi = { > > > }).done(function() { > > > url = 'https://' + location.hostname + ':' + proxy_port; > > > url += "/console.html?url=vnc_auto.html&port=" + > > proxy_port; > > > - url += "&path=?token=" + kimchi.urlSafeB64Encode(vm); > > > + /* > > > + * From python documentation base64.urlsafe_b64encode(s) > > > + * substitutes - instead of + and _ instead of / in the > > > + * standard Base64 alphabet, BUT the result can still > > > + * contain = which is not safe in a URL query component. > > > + * As token value is not decoded nowhere, replace = by A > > > + * */ > > > + url += "&path=?token=" + kimchi.urlSafeB64Encode > > (vm).replace(/=/g, 'A'); > > > url += "&kimchi=" + location.port; > > > url += '&encrypt=1'; > > > window.open(url); > > > @@ -377,7 +384,14 @@ var kimchi = { > > > url = 'https://' + location.hostname + ':' + proxy_port; > > > url += "/console.html?url=spice.html&port=" + proxy_port; > > > url += "&listen=" + location.hostname; > > > - url += "&token=" + kimchi.urlSafeB64Encode(vm); > > > + /* > > > + * From python documentation base64.urlsafe_b64encode(s) > > > + * substitutes - instead of + and _ instead of / in the > > > + * standard Base64 alphabet, BUT the result can still > > > + * contain = which is not safe in a URL query component. > > > + * As token value is not decoded nowhere, replace = by A > > > + * */ > > > + url += "&token=" + kimchi.urlSafeB64Encode > > (vm).replace(/=/g, 'A'); > > > url += "&kimchi=" + location.port; > > > url += '&encrypt=1'; > > > window.open(url); > > > > > > -- > > Thanks and best regards! > > > > Sheldon Feng(冯少合)<shaohef(a)linux.vnet.ibm.com> > > IBM Linux Technology Center > > > -- Thanks and best regards! Sheldon Feng(冯少合)<shaohef(a)linux.vnet.ibm.com> IBM Linux Technology Center

1 0

[PATCH] issue #389: Use 127.0.0.1 as VNC listener IP
by alinefm＠linux.vnet.ibm.com 28 Jul '14

28 Jul '14

From: Aline Manera <alinefm(a)linux.vnet.ibm.com> When setting the listener IP to 0.0.0.0 anyone can connect to the freshly configured kimchi virtual machine manager box on port 5900, 5901, etc. and take over any virtual machine, since VNC is per default configured by kimchi to listen on all IPs. As websockets proxy connects to QEMU-VNC from 127.0.0.1 also set VNC listener IP to 127.0.0.1 Signed-off-by: Aline Manera <alinefm(a)linux.vnet.ibm.com> --- src/kimchi/mockmodel.py | 2 +- src/kimchi/osinfo.py | 2 +- tests/test_mockmodel.py | 2 +- tests/test_model.py | 2 +- tests/test_rest.py | 6 +++--- tests/test_vmtemplate.py | 8 ++++---- 6 files changed, 11 insertions(+), 11 deletions(-) diff --git a/src/kimchi/mockmodel.py b/src/kimchi/mockmodel.py index 1584471..ed275d5 100644 --- a/src/kimchi/mockmodel.py +++ b/src/kimchi/mockmodel.py @@ -1064,7 +1064,7 @@ def __init__(self, uuid, name, template_info): 'memory': self.memory, 'cpus': self.cpus, 'icon': None, - 'graphics': {'type': 'vnc', 'listen': '0.0.0.0', + 'graphics': {'type': 'vnc', 'listen': '127.0.0.1', 'port': None}, 'users': ['user1', 'user2', 'root'], 'groups': ['group1', 'group2', 'admin'], diff --git a/src/kimchi/osinfo.py b/src/kimchi/osinfo.py index 39c9163..d2a1c1d 100644 --- a/src/kimchi/osinfo.py +++ b/src/kimchi/osinfo.py @@ -94,7 +94,7 @@ defaults = {'networks': ['default'], 'storagepool': '/storagepools/default', 'domain': 'kvm', 'arch': os.uname()[4], - 'graphics': {'type': 'vnc', 'listen': '0.0.0.0'}} + 'graphics': {'type': 'vnc', 'listen': '127.0.0.1'}} def _get_arch(): diff --git a/tests/test_mockmodel.py b/tests/test_mockmodel.py index 223879c..97b5fdb 100644 --- a/tests/test_mockmodel.py +++ b/tests/test_mockmodel.py @@ -154,7 +154,7 @@ def test_vm_info(self): self.assertEquals('images/icon-vm.png', info['icon']) self.assertEquals(stats_keys, set(info['stats'].keys())) self.assertEquals('vnc', info['graphics']['type']) - self.assertEquals('0.0.0.0', info['graphics']['listen']) + self.assertEquals('127.0.0.1', info['graphics']['listen']) def test_packages_update(self): pkgs = model.packagesupdate_get_list() diff --git a/tests/test_model.py b/tests/test_model.py index da9dae5..cab8288 100644 --- a/tests/test_model.py +++ b/tests/test_model.py @@ -118,7 +118,7 @@ def test_vm_graphics(self): info = inst.vm_lookup('kimchi-vnc') self.assertEquals('vnc', info['graphics']['type']) - self.assertEquals('0.0.0.0', info['graphics']['listen']) + self.assertEquals('127.0.0.1', info['graphics']['listen']) graphics = {'type': 'spice', 'listen': '127.0.0.1'} params = {'name': 'kimchi-spice', 'template': '/templates/test', diff --git a/tests/test_rest.py b/tests/test_rest.py index 3ea1927..4feb298 100644 --- a/tests/test_rest.py +++ b/tests/test_rest.py @@ -370,7 +370,7 @@ def test_vm_graphics(self): self.assertEquals(201, resp.status) # Verify the VM vm = json.loads(self.request('/vms/test-vm').read()) - self.assertEquals('0.0.0.0', vm['graphics']['listen']) + self.assertEquals('127.0.0.1', vm['graphics']['listen']) self.assertEquals('vnc', vm['graphics']['type']) # Delete the VM resp = self.request('/vms/test-vm', '{}', 'DELETE') @@ -412,7 +412,7 @@ def test_vm_graphics(self): self.assertEquals(201, resp.status) # Verify the VM vm = json.loads(self.request('/vms/test-vm').read()) - self.assertEquals('0.0.0.0', vm['graphics']['listen']) + self.assertEquals('127.0.0.1', vm['graphics']['listen']) self.assertEquals('spice', vm['graphics']['type']) # Delete the VM resp = self.request('/vms/test-vm', '{}', 'DELETE') @@ -1132,7 +1132,7 @@ def verify_template(t, res): # Update the template t['os_distro'] = 'Linux.ISO' t['os_version'] = '1.1' - t['graphics'] = {'type': 'vnc', 'listen': '0.0.0.0'} + t['graphics'] = {'type': 'vnc', 'listen': '127.0.0.1'} req = json.dumps(t) resp = self.request('/templates/%s' % t['name'], req, 'PUT') self.assertEquals(200, resp.status) diff --git a/tests/test_vmtemplate.py b/tests/test_vmtemplate.py index 821ca24..acf0150 100644 --- a/tests/test_vmtemplate.py +++ b/tests/test_vmtemplate.py @@ -31,7 +31,7 @@ def test_minimal_construct(self): ('os_version', 'unknown'), ('cpus', 1), ('memory', 1024), ('cdrom', ''), ('networks', ['default']), ('disk_bus', 'ide'), ('nic_model', 'e1000'), - ('graphics', {'type': 'vnc', 'listen': '0.0.0.0'})) + ('graphics', {'type': 'vnc', 'listen': '127.0.0.1'})) args = {'name': 'test'} t = VMTemplate(args) @@ -55,7 +55,7 @@ def test_specified_graphics(self): self.assertEquals(graphics, t.info['graphics']) # Test specified type - graphics = {'type': 'spice', 'listen': '0.0.0.0'} + graphics = {'type': 'spice', 'listen': '127.0.0.1'} args['graphics'] = graphics t = VMTemplate(args) self.assertEquals(graphics, t.info['graphics']) @@ -65,10 +65,10 @@ def test_specified_graphics(self): args['graphics'] = graphics t = VMTemplate(args) self.assertEquals(graphics['type'], t.info['graphics']['type']) - self.assertEquals('0.0.0.0', t.info['graphics']['listen']) + self.assertEquals('127.0.0.1', t.info['graphics']['listen']) def test_to_xml(self): - graphics = {'type': 'spice', 'listen': '0.0.0.0'} + graphics = {'type': 'spice', 'listen': '127.0.0.1'} vm_uuid = str(uuid.uuid4()).replace('-', '') t = VMTemplate({'name': 'test-template'}) xml = t.to_vm_xml('test-vm', vm_uuid, graphics=graphics) -- 1.9.3

4 3

[PATCH] bug fix: Avoid equals sign in VM console URL
by alinefm＠linux.vnet.ibm.com 28 Jul '14

28 Jul '14

From: Aline Manera <alinefm(a)linux.vnet.ibm.com> >From python documentation, base64.urlsafe_b64encode(s) substitutes - instead of + and _ instead of / in the standard Base64 alphabet, BUT the result can still contain = which is not safe in a URL query component. As token value is not decoded nowhere, replace = by A The problem with equals sign was only identified on Spice connections. noVNC can deal well with that. For reference: https://docs.python.org/2/library/base64.html Signed-off-by: Aline Manera <alinefm(a)linux.vnet.ibm.com> --- src/kimchi/vnc.py | 9 ++++++++- ui/js/src/kimchi.api.js | 18 ++++++++++++++++-- 2 files changed, 24 insertions(+), 3 deletions(-) diff --git a/src/kimchi/vnc.py b/src/kimchi/vnc.py index 9380e21..4159049 100644 --- a/src/kimchi/vnc.py +++ b/src/kimchi/vnc.py @@ -54,7 +54,14 @@ def new_ws_proxy(): def add_proxy_token(name, port): with open(os.path.join(WS_TOKENS_DIR, name), 'w') as f: - name = base64.urlsafe_b64encode(name) + """ + From python documentation base64.urlsafe_b64encode(s) + substitutes - instead of + and _ instead of / in the + standard Base64 alphabet, BUT the result can still + contain = which is not safe in a URL query component. + As token value is not decoded nowhere, replace = by A + """ + name = base64.urlsafe_b64encode(name).replace('=', 'A') f.write('%s: localhost:%s' % (name.encode('utf-8'), port)) diff --git a/ui/js/src/kimchi.api.js b/ui/js/src/kimchi.api.js index 8f5b68f..30360c5 100644 --- a/ui/js/src/kimchi.api.js +++ b/ui/js/src/kimchi.api.js @@ -352,7 +352,14 @@ var kimchi = { }).done(function() { url = 'https://' + location.hostname + ':' + proxy_port; url += "/console.html?url=vnc_auto.html&port=" + proxy_port; - url += "&path=?token=" + kimchi.urlSafeB64Encode(vm); + /* + * From python documentation base64.urlsafe_b64encode(s) + * substitutes - instead of + and _ instead of / in the + * standard Base64 alphabet, BUT the result can still + * contain = which is not safe in a URL query component. + * As token value is not decoded nowhere, replace = by A + * */ + url += "&path=?token=" + kimchi.urlSafeB64Encode(vm).replace(/=/g, 'A'); url += "&kimchi=" + location.port; url += '&encrypt=1'; window.open(url); @@ -377,7 +384,14 @@ var kimchi = { url = 'https://' + location.hostname + ':' + proxy_port; url += "/console.html?url=spice.html&port=" + proxy_port; url += "&listen=" + location.hostname; - url += "&token=" + kimchi.urlSafeB64Encode(vm); + /* + * From python documentation base64.urlsafe_b64encode(s) + * substitutes - instead of + and _ instead of / in the + * standard Base64 alphabet, BUT the result can still + * contain = which is not safe in a URL query component. + * As token value is not decoded nowhere, replace = by A + * */ + url += "&token=" + kimchi.urlSafeB64Encode(vm).replace(/=/g, 'A'); url += "&kimchi=" + location.port; url += '&encrypt=1'; window.open(url); -- 1.9.3

4 6

[PATCHv4 0/7] Support image based template
by lvroyce＠linux.vnet.ibm.com 28 Jul '14

28 Jul '14

From: Royce Lv <lvroyce(a)linux.vnet.ibm.com> v3>v4, Aggreated image scanning and name generate logic to vmtemplate.py to avoid duplicate code. Updated testcases accordingly. v2>v3, Clear unused iso link, Adding mockmodel and tests How to test: create a image using: POST /templates {'name':'mytemp', 'disks':[{'base':'a_base_img_path'}]} create a vm using: POST /vms {'template': '/templates/mytemp', 'pool'....} Passed test: 1. regression: (1) create vm from local iso based template (2) create vm from remote iso based template 2. name generating test: (1) iso/image with valid distro can produce right name (2) iso/image with unknown distro can produce right name 3. functional test: (1) create template from img (2) create vm from img based template Royce Lv (7): Add image probe function Change doc and api specification Change 'cdrom' to a optional param Fix: Prevent iso links filling in osinfo.py Create volume based on backing store image Update mockmodel of base img vm Add tests for image based template Makefile.am | 1 + contrib/DEBIAN/control.in | 4 +- contrib/kimchi.spec.fedora.in | 2 + contrib/kimchi.spec.suse.in | 2 + docs/API.md | 3 +- docs/README.md | 9 ++- src/kimchi/API.json | 8 ++- src/kimchi/control/storagevolumes.py | 2 +- src/kimchi/control/templates.py | 2 +- src/kimchi/exception.py | 4 ++ src/kimchi/i18n.py | 8 ++- src/kimchi/imageinfo.py | 66 ++++++++++++++++++++++ src/kimchi/mockmodel.py | 21 +++---- src/kimchi/model/templates.py | 12 +--- src/kimchi/model/vms.py | 1 + src/kimchi/osinfo.py | 24 +------- src/kimchi/vmtemplate.py | 104 +++++++++++++++++++++++++---------- tests/test_mockmodel.py | 12 ++-- tests/test_model.py | 25 +++++++++ tests/test_osinfo.py | 8 --- tests/test_rest.py | 35 +++++++++++- tests/test_vmtemplate.py | 25 ++++++--- 22 files changed, 276 insertions(+), 102 deletions(-) create mode 100644 src/kimchi/imageinfo.py -- 1.8.3.2

3 10

[PATCH] Do not auto create default network
by alinefm＠linux.vnet.ibm.com 28 Jul '14

28 Jul '14

From: Aline Manera <alinefm(a)linux.vnet.ibm.com> Kimchi does not depends on default network as a template and a guest can be created without any network interface. So do not auto create the default network when starting up Kimchi and let users create and configure their networks and templates as they want. It is part of issue #392 Signed-off-by: Aline Manera <alinefm(a)linux.vnet.ibm.com> --- src/kimchi/mockmodel.py | 8 +------- src/kimchi/model/networks.py | 44 -------------------------------------------- src/kimchi/osinfo.py | 2 +- tests/test_model.py | 19 +++++++------------ tests/test_osinfo.py | 2 +- tests/test_rest.py | 19 +++++-------------- tests/test_vmtemplate.py | 2 +- 7 files changed, 16 insertions(+), 80 deletions(-) diff --git a/src/kimchi/mockmodel.py b/src/kimchi/mockmodel.py index 1584471..2a74044 100644 --- a/src/kimchi/mockmodel.py +++ b/src/kimchi/mockmodel.py @@ -81,7 +81,7 @@ def reset(self): self._mock_screenshots = {} self._mock_templates = {} self._mock_storagepools = {'default': MockStoragePool('default')} - self._mock_networks = {'default': MockNetwork('default')} + self._mock_networks = {} self._mock_interfaces = self.dummy_interfaces() self._mock_swupdate = MockSoftwareUpdate() self.next_taskid = 1 @@ -659,12 +659,6 @@ def _is_network_used_by_template(self, network): return False def _is_network_in_use(self, name): - # The network "default" is used for Kimchi proposal and should not be - # deactivate or deleted. Otherwise, we will allow user create - # inconsistent templates from scratch - if name == 'default': - return True - vms = self._get_vms_attach_to_a_network(name) return bool(vms) or self._is_network_used_by_template(name) diff --git a/src/kimchi/model/networks.py b/src/kimchi/model/networks.py index bc93da5..45659ea 100644 --- a/src/kimchi/model/networks.py +++ b/src/kimchi/model/networks.py @@ -41,44 +41,6 @@ class NetworksModel(object): def __init__(self, **kargs): self.conn = kargs['conn'] - if 'qemu:///' in self.conn.get().getURI(): - self._default_network_check() - - def _default_network_check(self): - def create_default_network(): - try: - subnet = self._get_available_address(knetwork.DefaultNetsPool) - params = {"name": "default", "connection": "nat", - "subnet": subnet} - self.create(params) - return conn.networkLookupByName("default") - except Exception as e: - kimchi_log.error("Fatal: Cannot create default network " - "because of %s, exit kimchid", e.message) - sys.exit(1) - - conn = self.conn.get() - try: - net = conn.networkLookupByName("default") - except libvirt.libvirtError: - net = create_default_network() - - if net.isActive() == 0: - try: - net.create() - except libvirt.libvirtError as e: - # FIXME we can not distinguish this error from other internal - # error by error code. - if ("network is already in use by interface" - in e.message.lower()): - # libvirt do not support update IP element, so delete the - # the network and create new one. - net.undefine() - create_default_network() - else: - kimchi_log.error("Fatal: Cannot activate default network " - "because of %s, exit kimchid", e.message) - sys.exit(1) def create(self, params): conn = self.conn.get() @@ -267,12 +229,6 @@ def lookup(self, name): 'persistent': True if network.isPersistent() else False} def _is_network_in_use(self, name): - # The network "default" is used for Kimchi proposal and should not be - # deactivate or deleted. Otherwise, we will allow user create - # inconsistent templates from scratch - if name == 'default': - return True - vms = self._get_vms_attach_to_a_network(name) return bool(vms) or self._is_network_used_by_template(name) diff --git a/src/kimchi/osinfo.py b/src/kimchi/osinfo.py index 39c9163..cec2c50 100644 --- a/src/kimchi/osinfo.py +++ b/src/kimchi/osinfo.py @@ -91,7 +91,7 @@ }, } -defaults = {'networks': ['default'], +defaults = {'networks': [], 'storagepool': '/storagepools/default', 'domain': 'kvm', 'arch': os.uname()[4], 'graphics': {'type': 'vnc', 'listen': '0.0.0.0'}} diff --git a/tests/test_model.py b/tests/test_model.py index da9dae5..e717872 100644 --- a/tests/test_model.py +++ b/tests/test_model.py @@ -154,12 +154,7 @@ def test_vm_ifaces(self): rollback.prependDefer(inst.network_deactivate, net_name) ifaces = inst.vmifaces_get_list('kimchi-ifaces') - self.assertEquals(1, len(ifaces)) - - iface = inst.vmiface_lookup('kimchi-ifaces', ifaces[0]) - self.assertEquals(17, len(iface['mac'])) - self.assertEquals("default", iface['network']) - self.assertIn("model", iface) + self.assertEquals(0, len(ifaces)) # attach network interface to vm iface_args = {"type": "network", @@ -176,11 +171,11 @@ def test_vm_ifaces(self): self.assertEquals("virtio", iface["model"]) # update vm interface - iface_args = {"network": "default", + iface_args = {"network": "test-network", "model": "e1000"} inst.vmiface_update('kimchi-ifaces', mac, iface_args) iface = inst.vmiface_lookup('kimchi-ifaces', mac) - self.assertEquals("default", iface['network']) + self.assertEquals("test-network", iface['network']) self.assertEquals("e1000", iface["model"]) @unittest.skipUnless(utils.running_as_root(), 'Must be run as root') @@ -565,14 +560,14 @@ def test_template_create(self): info = inst.template_lookup('test') for key in params.keys(): self.assertEquals(params[key], info[key]) - self.assertEquals("default", info["networks"][0]) + self.assertEquals(0, len(info["networks"])) # create template with non-existent network params['name'] = 'new-test' params['networks'] = ["no-exist"] self.assertRaises(InvalidParameter, inst.templates_create, params) - params['networks'] = ['default', 'test-network'] + params['networks'] = ['test-network'] inst.templates_create(params) rollback.prependDefer(inst.template_delete, params['name']) info = inst.template_lookup(params['name']) @@ -667,10 +662,10 @@ def test_template_update(self): info = inst.template_lookup('new-test') for key in params.keys(): self.assertEquals(params[key], info[key]) - self.assertEquals("default", info["networks"][0]) + self.assertEquals(0, len(info["networks"])) params = {'name': 'new-test', 'memory': 1024, 'cpus': 1, - 'networks': ['default', 'test-network']} + 'networks': ['test-network']} inst.template_update('new-test', params) info = inst.template_lookup('new-test') for key in params.keys(): diff --git a/tests/test_osinfo.py b/tests/test_osinfo.py index 104e7b8..7524f84 100644 --- a/tests/test_osinfo.py +++ b/tests/test_osinfo.py @@ -28,7 +28,7 @@ def test_default_lookup(self): entry = lookup(None, None) self.assertEquals('unknown', entry['os_distro']) self.assertEquals('unknown', entry['os_version']) - self.assertEquals(['default'], entry['networks']) + self.assertEquals([], entry['networks']) def test_fedora_lookup(self): cd = ('http://fedora.mirrors.tds.net/pub/fedora/releases/17/Live/' diff --git a/tests/test_rest.py b/tests/test_rest.py index 3ea1927..306064c 100644 --- a/tests/test_rest.py +++ b/tests/test_rest.py @@ -603,14 +603,7 @@ def test_vm_iface(self): '/networks/test-network', '{}', 'DELETE') ifaces = json.loads(self.request('/vms/test-vm/ifaces').read()) - self.assertEquals(1, len(ifaces)) - - for iface in ifaces: - res = json.loads(self.request('/vms/test-vm/ifaces/%s' % - iface['mac']).read()) - self.assertEquals('default', res['network']) - self.assertEquals(17, len(res['mac'])) - self.assertEquals('virtio', res['model']) + self.assertEquals(0, len(ifaces)) # attach network interface to vm req = json.dumps({"type": "network", @@ -794,7 +787,7 @@ def test_template_customise_network(self): '/templates/test', '{}', 'DELETE') tmpl_res = json.loads(resp.read()) self.assertTrue(type(tmpl_res['networks']) is list) - self.assertEquals("default", tmpl_res['networks'][0]) + self.assertEquals(0, len(tmpl_res['networks'])) tmpl['name'] = "failed_tmpl" # Create a Template with non-array network fails with 400 @@ -832,7 +825,7 @@ def test_template_customise_network(self): self.assertEquals(400, resp.status) # Update a Template with existent network, successful - tmpl['networks'] = ["default", "test-network"] + tmpl['networks'] = ["test-network"] req = json.dumps(tmpl) resp = self.request('/templates/test', req, 'PUT') self.assertEquals(200, resp.status) @@ -1407,9 +1400,7 @@ def test_interfaces(self): def test_get_networks(self): networks = json.loads(request(host, ssl_port, '/networks').read()) - self.assertEquals(1, len(networks)) - self.assertEquals('default', networks[0]['name']) - self.assertEquals([], networks[0]['vms']) + self.assertEquals(0, len(networks)) # Now add a couple of Networks to the mock model for i in xrange(5): @@ -1424,7 +1415,7 @@ def test_get_networks(self): self.assertEquals([], network["vms"]) networks = json.loads(request(host, ssl_port, '/networks').read()) - self.assertEquals(6, len(networks)) + self.assertEquals(5, len(networks)) network = json.loads(request(host, ssl_port, '/networks/network-1').read()) diff --git a/tests/test_vmtemplate.py b/tests/test_vmtemplate.py index 821ca24..8dceced 100644 --- a/tests/test_vmtemplate.py +++ b/tests/test_vmtemplate.py @@ -29,7 +29,7 @@ class VMTemplateTests(unittest.TestCase): def test_minimal_construct(self): fields = (('name', 'test'), ('os_distro', 'unknown'), ('os_version', 'unknown'), ('cpus', 1), - ('memory', 1024), ('cdrom', ''), ('networks', ['default']), + ('memory', 1024), ('cdrom', ''), ('networks', []), ('disk_bus', 'ide'), ('nic_model', 'e1000'), ('graphics', {'type': 'vnc', 'listen': '0.0.0.0'})) -- 1.9.3

4 3

[PATCH] PowerPC bootable ISO detection code
by Daniel Barboza 27 Jul '14

27 Jul '14

From: Daniel Henrique Barboza <danielhb(a)linux.vnet.ibm.com> PowerPC ISO images does not follow the same rules as Intel's. Intel ISOs uses ISO9660 extension called 'El Torito', which describes the format of the Boot Record. PowerPC firmware, on the other hand, looks for a file '/ppc/bootinfo.txt' which describes how to boot the image. This patch creates a function that opens the ISO file without mounting it and, following ISO9660 standards, read the filesystem trying to find the boot file that enables PowerPC booting. Signed-off-by: Daniel Henrique Barboza <danielhb(a)linux.vnet.ibm.com> --- src/kimchi/isoinfo.py | 166 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 165 insertions(+), 1 deletion(-) diff --git a/src/kimchi/isoinfo.py b/src/kimchi/isoinfo.py index c394a32..72b956d 100644 --- a/src/kimchi/isoinfo.py +++ b/src/kimchi/isoinfo.py @@ -18,6 +18,7 @@ # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA import glob +import platform import os import re import struct @@ -132,6 +133,12 @@ class IsoImage(object): EL_TORITO_BOOT_RECORD = struct.Struct("=B5sB32s32sI") EL_TORITO_VALIDATION_ENTRY = struct.Struct("=BBH24sHBB") EL_TORITO_BOOT_ENTRY = struct.Struct("=BBHBBHL20x") + # Path table info starting in ISO9660 offset 132. We force little + # endian byte order (the '<' sign) because Power systems can run on + # both. + # First int is path table size, next 4 bytes are discarded (it is + # the same info but in big endian) and next int is the location. + PATH_TABLE_SIZE_LOC = struct.Struct("<I 4s I") def __init__(self, path): self.path = path @@ -224,6 +231,160 @@ class IsoImage(object): raise IsoFormatError("KCHISO0005E", {'filename': self.path}) + def _scan_ppc(self): + """ + PowerPC firmware does not use the conventional El Torito boot + specification. Instead, it looks for a file '/ppc/bootinfo.txt' + which contains boot information. A PPC image is bootable if + this file exists in the filesystem [1]. + + To detect if a PPC ISO is bootable, we could simply mount the + ISO and search for the boot file as we would with any other + file in the filesystem. We can also look for the boot file + searching byte by byte the ISO image. This is possible because + the PPC ISO image follows the ISO9660 standard [2]. Mounting + the ISO requires extra resources and it takes longer than + searching the image data, thus we chose the latter approach + in this code. + + To locate a file we must access the Path Table, which contains + the records of all the directories in the ISO. After locating + the directory/subdirectory that contains the file, we access + the Directory Record to find it. + + + .. [1] https://www.ibm.com/developerworks/community/wikis/home?\ +lang=en#!/wiki/W51a7ffcf4dfd_4b40_9d82_446ebc23c550/page/PowerLinux\ +%20Boot%20howto + .. [2] http://wiki.osdev.org/ISO_9660 + """ + + # To locate any file we must access the Path Table, which + # contains the records of all the directories in the ISO. + # ISO9660 dictates that the Path Table location information + # is at offset 132, inside the Primary Volume Descriptor, + # after the SystemArea (16*SECTOR_SIZE). + # + # In the Path table info we're forcing little endian byte + # order (the '<' sign) because Power systems can run on + # both. + # + # First int is path table size, next 4 bytes are discarded (it is + # the same info but in big endian) and next int is the location. + PATH_TABLE_LOC_OFFSET = 16 * IsoImage.SECTOR_SIZE + 132 + PATH_TABLE_SIZE_LOC = struct.Struct("<I 4s I") + + path_table_loc_data = self._get_iso_data(PATH_TABLE_LOC_OFFSET, + PATH_TABLE_SIZE_LOC.size) + path_size, unused, path_loc = self._unpack(PATH_TABLE_SIZE_LOC, + path_table_loc_data) + # Fetch the Path Table using location and size found above + path_table_offset = path_loc * IsoImage.SECTOR_SIZE + path_table_data = self._get_iso_data(path_table_offset, path_size) + + # Loop inside the path table to find the directory 'ppc'. + # The contents of the registers are: + # - length of the directory identifier (1 byte) + # - extended attribute record length (1 byte) + # - location of directory register (4 bytes) + # - directory number of parent dir (2 bytes) + # - directory name (size varies according to length) + # - padding field - 1 byte if the length is odd, not present if even + DIR_NAMELEN_LOCATION_PARENT = struct.Struct("<B B I H") + dir_struct_size = DIR_NAMELEN_LOCATION_PARENT.size + i = 0 + while i < path_size: + dir_data = path_table_data[i: i+dir_struct_size] + i += dir_struct_size + # We won't use the Extended Attribute Record + dir_namelen, unused, dir_loc, dir_parent = \ + self._unpack(DIR_NAMELEN_LOCATION_PARENT, dir_data) + if dir_parent == 1: + # read the dir name using the namelen + dir_name = path_table_data[i: i+dir_namelen].rstrip() + if dir_name == 'ppc': + # stop searching, dir was found + break + # Need to consider the optional padding field as well + i += dir_namelen + dir_namelen % 2 + + if i > path_size: + # Didn't find the '/ppc' directory. ISO is not bootable. + self.bootable = False + return + + # Get the 'ppc' directory record using 'dir_loc'. + ppc_dir_offset = dir_loc * IsoImage.SECTOR_SIZE + + # We need to find the sector size of this dir entry. The + # size of the File Section is located 10 bytes after + # the dir location. + DIR_SIZE_FMT = struct.Struct("<10sI") + data = self._get_iso_data(ppc_dir_offset, DIR_SIZE_FMT.size) + unused, dir_size = self._unpack(DIR_SIZE_FMT, data) + # If the dir is in the middle of a sector, the sector is + # padded zero and won't be utilized. We need to round up + # the result + dir_sectorsize = dir_size / IsoImage.SECTOR_SIZE + if dir_size % IsoImage.SECTOR_SIZE: + dir_sectorsize += 1 + + # Fixed-size directory record fields: + # - length of directory record (1 byte) + # - extended attr. record length (1 byte) + # - location of extend in both-endian format (8 bytes) + # - data length (size of extend) in both-endian (8 bytes) + # - recording date and time (7 bytes) + # - file flags (1 byte) + # - file unit size interleaved (1 byte) + # - interleave gap size (1 byte) + # - volume sequence number (4 bytes) + # - length of file identifier (1 byte) + # + # Of all these fields, we will use only 3 of them, 'ignoring' + # 30 bytes total. + STATIC_DIR_RECORD_FMT = struct.Struct("<B 24s B 6s B") + static_rec_size = STATIC_DIR_RECORD_FMT.size + + # Maximum offset possible of all the records of this directory + DIR_REC_MAX = ppc_dir_offset + dir_sectorsize*IsoImage.SECTOR_SIZE + # Max size of a given directory record + MAX_DIR_SIZE = 255 + # Name of the boot file + BOOT_FILE_NAME = "bootinfo.txt" + + # Loop until one of the following happens: + # - boot file is found + # - end of directory record listing for the 'ppc' dir + while ppc_dir_offset < DIR_REC_MAX: + record_data = self._get_iso_data(ppc_dir_offset, MAX_DIR_SIZE) + dir_rec_len, unused, file_flags, unused2, file_name_len = \ + self._unpack(STATIC_DIR_RECORD_FMT, record_data) + + # if dir_rec_len = 0, increment offset (skip the + # dir_rec_len byte) and continue the loop + if dir_rec_len == 0: + ppc_dir_offset += 1 + continue + + # Get filename of the file/dir we're at. + filename = record_data[static_rec_size: + static_rec_size + file_name_len].rstrip() + # The second bit of the file_flags indicate if this record + # is a directory. + if filename == BOOT_FILE_NAME and (file_flags & 2) != 1: + self.bootable = True + return + + # Update offset and keep looking. There is a padding here + # if the length of the file identifier is EVEN. + padding = 0 + if not file_name_len % 2: + padding = 1 + ppc_dir_offset += dir_rec_len + padding + # If reached this point the file wasn't found = not bootable + self.bootable = False + def _scan_primary_vol(self, data): """ Scan one sector for a Primary Volume Descriptor and extract the @@ -269,7 +430,10 @@ class IsoImage(object): return self._scan_primary_vol(data) - self._scan_el_torito(data) + if platform.machine().startswith('ppc'): + self._scan_ppc() + else: + self._scan_el_torito(data) class Matcher(object): -- 1.8.3.1

1 0

[V2 0/2] Authorization: Add users/groups to VM
by huoyuxin＠linux.vnet.ibm.com 25 Jul '14

25 Jul '14

From: Yu Xin Huo <huoyuxin(a)linux.vnet.ibm.com> Yu Xin Huo (2): Add users/groups to VM po files update for permission po/en_US.po | 9 +++ po/kimchi.pot | 9 +++ po/pt_BR.po | 9 +++ po/zh_CN.po | 9 +++ ui/css/theme-default/guest-edit.css | 116 +++++++++++++++++++++++++++++++++++ ui/images/theme-default/group.png | Bin 0 -> 1703 bytes ui/images/theme-default/user.png | Bin 0 -> 1322 bytes ui/js/src/kimchi.api.js | 28 ++++++++ ui/js/src/kimchi.guest_edit_main.js | 112 ++++++++++++++++++++++++++++++++- ui/pages/guest-edit.html.tmpl | 37 +++++++++++ 10 files changed, 326 insertions(+), 3 deletions(-) create mode 100644 ui/images/theme-default/group.png create mode 100644 ui/images/theme-default/user.png

3 7

[PATCH 0/9] authorization: Filter resources by users and groups
by alinefm＠linux.vnet.ibm.com 24 Jul '14

24 Jul '14

From: Aline Manera <alinefm(a)linux.vnet.ibm.com> Also move autorization mechanism to controller to be able to distinguish resource and collection configuration If we use UrlSubNode to also handle the authorization configuration, we won't be able to specify different configuration to collection and its resource as Kimchi uses the same base URL to both. Example: @UrlSubNode("vms", True, ["POST", "PUT", "DELETE"], 'guests') It meant that all the methods listed were exclusive for admin users. Which it is not correct, as a user assigned to a VM can also perform POST, PUT and DELETE actions. So fix it by moving the authorization mechanism to controller Aline Manera (5): authorization: Filter resources by users and groups authorization: Restrict Collection access based on admin_methods parameter authorization: Restrict access to Resource instance authorization: Update control files to set role_key and admin_methods authorization: Remove authorization config from UrlSubNode Crístian Viana (4): Return some groups for every user in mockmodel Move "fake_user" credentials to mockmodel List "admin" as a valid system user in mockmodel authorization: Update test cases based on last changes src/kimchi/auth.py | 16 +---------- src/kimchi/control/base.py | 56 +++++++++++++++++++++++++++++++----- src/kimchi/control/debugreports.py | 8 +++++- src/kimchi/control/host.py | 26 +++++++++++++++-- src/kimchi/control/interfaces.py | 6 +++- src/kimchi/control/networks.py | 6 +++- src/kimchi/control/storagepools.py | 6 +++- src/kimchi/control/storageservers.py | 8 +++++- src/kimchi/control/templates.py | 6 +++- src/kimchi/control/utils.py | 14 +++++---- src/kimchi/control/vms.py | 6 +++- src/kimchi/exception.py | 4 +++ src/kimchi/i18n.py | 1 + src/kimchi/mockmodel.py | 5 +++- src/kimchi/server.py | 4 --- tests/test_authorization.py | 30 +++++++++++++++++-- tests/test_rest.py | 19 ++++++------ tests/utils.py | 9 +++--- 18 files changed, 172 insertions(+), 58 deletions(-) -- 1.9.3

3 11

[V3 0/2] Authorization: Add Users/Groups to VM
by huoyuxin＠linux.vnet.ibm.com 24 Jul '14

24 Jul '14

From: Yu Xin Huo <huoyuxin(a)linux.vnet.ibm.com> Yu Xin Huo (2): authorization: add users/groups to vm authorization: update po files po/en_US.po | 9 +++ po/kimchi.pot | 9 +++ po/pt_BR.po | 9 +++ po/zh_CN.po | 9 +++ ui/css/theme-default/guest-edit.css | 104 ++++++++++++++++++++++++++++++ ui/images/theme-default/group.png | Bin 0 -> 1703 bytes ui/images/theme-default/user.png | Bin 0 -> 1322 bytes ui/js/src/kimchi.api.js | 28 ++++++++ ui/js/src/kimchi.guest_edit_main.js | 120 ++++++++++++++++++++++++++++++++++- ui/pages/guest-edit.html.tmpl | 42 ++++++++++++ 10 files changed, 327 insertions(+), 3 deletions(-) create mode 100644 ui/images/theme-default/group.png create mode 100644 ui/images/theme-default/user.png

2 5