The URI /host/devices only presents scsi_host (particularly fc_host) device information. To implement host PCI pass through, we should list all types of host devices. This patch adds support for parsing various host devices information, and listing them on /host/devices. So the user is free to choose any listed PCI device to pass through to guest. Since the patch changes the device information dictionary format, the existing code consuming the device information is also changed accordingly. To get all types of host device, access the following URL. curl -k -u root -H "Content-Type: application/json" \ -H "Accept: application/json" \ https://127.0.0.1:8001/host/devices To get only fc_host devices, change the URL to "https://127.0.0.1:8001/host/devices?_cap=fc_host" To get only pci device, change the URL to "https://127.0.0.1:8001/host/devices?_cap=pci" v1: Parse the node device XML using xpath. v2: Write a "dictize" function and parse the node device XML using dictize. v3: Fix a naming mistake. v4: It is observed that sometimes the parent devices is not listed by libvirt but the child device is listed. In previous version we catch this exception and ignore it. The root cause is unknown, and we failed to re-produce the problem. In v4 we do not catch it. It seems to be related to USB removable disk, and the problem is gone after we upgraded Linux kernel. v8: Move hostdev.py from src/kimchi to src/kimchi/model. v9: Improve Ubuntu and Fedora 19 compatibility. Gather device information if libvirt does not provide enough information. Share the same LibvirtConnection object with the Model class, to prevent connection exhausting. v10: Adapt to RHEL 6. RHEL 6 does not provide iommu group information in sysfs. For now we just ignore this error and live with it. The device passthrough for PCI devices will not work, but the basic devices informations are still provided to the user. In future we'll develope code to gather iommu group information. v11: Properly Parse NPIV Capable HBA Card Information. v12: Coding style improvement. Use reflection to avoid maintaining function map. Signed-off-by: Zhou Zheng Sheng <zhshzhou(a)linux.vnet.ibm.com&gt; --- docs/API.md | 11 +- src/kimchi/mockmodel.py | 7 +- src/kimchi/model/host.py | 15 +-- src/kimchi/model/hostdev.py | 210 +++++++++++++++++++++++++++++++++ src/kimchi/model/libvirtstoragepool.py | 18 +-- src/kimchi/xmlutils.py | 24 ++++ tests/test_rest.py | 6 +- tests/test_storagepool.py | 7 +- 8 files changed, 262 insertions(+), 36 deletions(-) create mode 100644 src/kimchi/model/hostdev.py diff --git a/docs/API.md b/docs/API.md index cc438cc..b65f211 100644 --- a/docs/API.md +++ b/docs/API.md @@ -912,12 +912,11 @@ stats history * **GET**: Retrieve information of a single pci device. Currently only scsi_host devices are supported: * name: The name of the device. - * adapter_type: The capability type of the scsi_host device (fc_host). - Empty if pci device is not scsi_host. - * wwnn: The HBA Word Wide Node Name. - Empty if pci device is not scsi_host. - * wwpn: The HBA Word Wide Port Name - Empty if pci device is not scsi_host. + * path: Path of device in sysfs. + * adapter: Host adapter information. Empty if pci device is not scsi_host. + * type: The capability type of the scsi_host device (fc_host, vport_ops). + * wwnn: The HBA Word Wide Node Name. Empty if pci device is not fc_host. + * wwpn: The HBA Word Wide Port Name. Empty if pci device is not fc_host. ### Collection: Host Packages Update diff --git a/src/kimchi/mockmodel.py b/src/kimchi/mockmodel.py index cbbdba3..0fa16e8 100644 --- a/src/kimchi/mockmodel.py +++ b/src/kimchi/mockmodel.py @@ -617,9 +617,10 @@ class MockModel(object): def device_lookup(self, nodedev_name): return { 'name': nodedev_name, - 'adapter_type': 'fc_host', - 'wwnn': uuid.uuid4().hex[:16], - 'wwpn': uuid.uuid4().hex[:16]} + 'adapter': { + 'type': 'fc_host', + 'wwnn': uuid.uuid4().hex[:16], + 'wwpn': uuid.uuid4().hex[:16]}} def isopool_lookup(self, name): return {'state': 'active', diff --git a/src/kimchi/model/host.py b/src/kimchi/model/host.py index 3975a10..7d7cd66 100644 --- a/src/kimchi/model/host.py +++ b/src/kimchi/model/host.py @@ -33,6 +33,7 @@ from kimchi import netinfo from kimchi import xmlutils from kimchi.basemodel import Singleton from kimchi.exception import InvalidOperation, NotFoundError, OperationFailed +from kimchi.model import hostdev from kimchi.model.config import CapabilitiesModel from kimchi.model.tasks import TaskModel from kimchi.model.vms import DOM_STATE_MAP @@ -341,20 +342,10 @@ class DeviceModel(object): def lookup(self, nodedev_name): conn = self.conn.get() try: - dev_xml = conn.nodeDeviceLookupByName(nodedev_name).XMLDesc(0) + dev = conn.nodeDeviceLookupByName(nodedev_name) except: raise NotFoundError('KCHHOST0003E', {'name': nodedev_name}) - cap_type = xmlutils.xpath_get_text( - dev_xml, '/device/capability/capability/@type') - wwnn = xmlutils.xpath_get_text( - dev_xml, '/device/capability/capability/wwnn') - wwpn = xmlutils.xpath_get_text( - dev_xml, '/device/capability/capability/wwpn') - return { - 'name': nodedev_name, - 'adapter_type': cap_type[0] if len(cap_type) >= 1 else '', - 'wwnn': wwnn[0] if len(wwnn) == 1 else '', - 'wwpn': wwpn[0] if len(wwpn) == 1 else ''} + return hostdev.get_dev_info(dev) class PackagesUpdateModel(object): diff --git a/src/kimchi/model/hostdev.py b/src/kimchi/model/hostdev.py new file mode 100644 index 0000000..103c1e7 --- /dev/null +++ b/src/kimchi/model/hostdev.py @@ -0,0 +1,210 @@ +# +# Kimchi +# +# Copyright IBM Corp, 2014 +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +from pprint import pformat + +from kimchi.model.libvirtconnection import LibvirtConnection +from kimchi.utils import kimchi_log +from kimchi.xmlutils import dictize + + +def _get_all_host_dev_infos(libvirt_conn): + node_devs = libvirt_conn.listAllDevices(0) + return [get_dev_info(node_dev) for node_dev in node_devs] + + +def _get_dev_info_tree(dev_infos): + devs = dict([(dev_info['name'], dev_info) for dev_info in dev_infos]) + root = None + for dev_info in dev_infos: + if dev_info['parent'] is None: + root = dev_info + continue + parent = devs[dev_info['parent']] + + try: + children = parent['children'] + except KeyError: + parent['children'] = [dev_info] + else: + children.append(dev_info) + return root + + +def get_dev_info(node_dev): + ''' Parse the node device XML string into dict according to + http://libvirt.org/formatnode.html. + + scsi_generic is not documented in libvirt official website. Try to + parse scsi_generic according to the following libvirt path series. + https://www.redhat.com/archives/libvir-list/2013-June/msg00014.html + + scsi_target is not documented in libvirt official website. Try to + parse scsi_target according to the libvirt commit db19834a0a. + ''' + + xmlstr = node_dev.XMLDesc(0) + info = dictize(xmlstr)['device'] + dev_type = info['capability'].pop('type') + info['device_type'] = dev_type + cap_dict = info.pop('capability') + info.update(cap_dict) + info['parent'] = node_dev.parent() + + if dev_type in ('scsi', 'scsi_generic', 'scsi_target', 'system', 'usb'): + return info + + if dev_type in ('net', 'pci', 'scsi_host', 'storage', 'usb_device'): + return globals()['_get_%s_dev_info' % dev_type](info) + + kimchi_log.error("Unknown device type: %s", dev_type) + return info + + +def _get_net_dev_info(info): + cap = info.pop('capability') + links = {"80203": "IEEE 802.3", "80211": "IEEE 802.11"} + link_raw = cap['type'] + info['link_type'] = links.get(link_raw, link_raw) + + return info + + +def _get_pci_dev_info(info): + for k in ('vendor', 'product'): + try: + description = info[k].pop('pyval') + except KeyError: + description = None + info[k]['description'] = description + if 'path' not in info: + # Old libvirt does not provide syspath info + info['path'] = \ + "/sys/bus/pci/devices/" \ + "%(domain)04x:%(bus)02x:%(slot)02x.%(function)01x" % { + 'domain': info['domain'], 'bus': info['bus'], + 'slot': info['slot'], 'function': info['function']} + try: + info['iommuGroup'] = int(info['iommuGroup']['number']) + except KeyError: + # Old libvirt does not provide syspath info, figure it out ourselves + iommu_link = os.path.join(info['path'], 'iommu_group') + if os.path.exists(iommu_link): + iommu_path = os.path.realpath(iommu_link) + try: + info['iommuGroup'] = int(iommu_path.rsplit('/', 1)[1]) + except (ValueError, IndexError): + # No IOMMU group support at all. + pass + else: + # No IOMMU group support at all. + pass + return info + + +def _get_scsi_host_dev_info(info): + try: + cap_info = info.pop('capability') + except KeyError: + # kimchi.model.libvirtstoragepool.ScsiPoolDef assumes + # info['adapter']['type'] always exists. + info['adapter'] = {'type': ''} + return info + if isinstance(cap_info, list): + info['adapter'] = {} + for cap in cap_info: + if cap['type'] == 'vport_ops': + del cap['type'] + info['adapter']['vport_ops'] = cap + else: + info['adapter'].update(cap) + else: + info['adapter'] = cap_info + return info + + +def _get_storage_dev_info(info): + try: + cap_info = info.pop('capability') + except KeyError: + return info + + if cap_info['type'] == 'removable': + cap_info['available'] = bool(cap_info.pop('media_available')) + if cap_info['available']: + for k in ('size', 'label'): + try: + cap_info[k] = cap_info.pop('media_' + k) + except KeyError: + cap_info[k] = None + info['media'] = cap_info + return info + + +def _get_usb_device_dev_info(info): + for k in ('vendor', 'product'): + try: + info[k]['description'] = info[k].pop('pyval') + except KeyError: + # Some USB devices don't provide vendor/product description. + pass + return info + + +# For test and debug +def _print_host_dev_tree(): + libvirt_conn = LibvirtConnection('qemu:///system').get() + dev_infos = _get_all_host_dev_infos(libvirt_conn) + root = _get_dev_info_tree(dev_infos) + if root is None: + print "No device found" + return + print '-----------------' + print '\n'.join(_format_dev_node(root)) + + +def _format_dev_node(node): + try: + children = node['children'] + del node['children'] + except KeyError: + children = [] + + lines = [] + lines.extend([' ~' + line for line in pformat(node).split('\n')]) + + count = len(children) + for i, child in enumerate(children): + if count == 1: + lines.append(' \-----------------') + else: + lines.append(' +-----------------') + clines = _format_dev_node(child) + if i == count - 1: + p = ' ' + else: + p = ' |' + lines.extend([p + cline for cline in clines]) + lines.append('') + + return lines + + +if __name__ == '__main__': + _print_host_dev_tree() diff --git a/src/kimchi/model/libvirtstoragepool.py b/src/kimchi/model/libvirtstoragepool.py index d39835b..d7b49e2 100644 --- a/src/kimchi/model/libvirtstoragepool.py +++ b/src/kimchi/model/libvirtstoragepool.py @@ -180,34 +180,34 @@ class ScsiPoolDef(StoragePoolDef): self.poolArgs['source']['name'] = tmp_name.replace('scsi_', '') # fc_host adapters type are only available in libvirt >= 1.0.5 if not self.poolArgs['fc_host_support']: - self.poolArgs['source']['adapter_type'] = 'scsi_host' + self.poolArgs['source']['adapter']['type'] = 'scsi_host' msg = "Libvirt version <= 1.0.5. Setting SCSI host name as '%s'; "\ "setting SCSI adapter type as 'scsi_host'; "\ "ignoring wwnn and wwpn." % tmp_name kimchi_log.info(msg) # Path for Fibre Channel scsi hosts self.poolArgs['path'] = '/dev/disk/by-path' - if not self.poolArgs['source']['adapter_type']: - self.poolArgs['source']['adapter_type'] = 'scsi_host' + if not self.poolArgs['source']['adapter']['type']: + self.poolArgs['source']['adapter']['type'] = 'scsi_host' @property def xml(self): # Required parameters # name: - # source[adapter_type]: + # source[adapter][type]: # source[name]: - # source[wwnn]: - # source[wwpn]: + # source[adapter][wwnn]: + # source[adapter][wwpn]: # path: xml = """ <pool type='scsi'> <name>{name}</name> <source> - <adapter type='{source[adapter_type]}'\ + <adapter type='{source[adapter][type]}'\ name='{source[name]}'\ - wwnn='{source[wwnn]}'\ - wwpn='{source[wwpn]}'/> + wwnn='{source[adapter][wwnn]}'\ + wwpn='{source[adapter][wwpn]}'/> </source> <target> <path>{path}</path> diff --git a/src/kimchi/xmlutils.py b/src/kimchi/xmlutils.py index d3db32a..00a9d55 100644 --- a/src/kimchi/xmlutils.py +++ b/src/kimchi/xmlutils.py @@ -18,6 +18,7 @@ # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA import libxml2 +from lxml import objectify from xml.etree import ElementTree @@ -37,3 +38,26 @@ def xml_item_update(xml, xpath, value): item = root.find(xpath) item.text = value return ElementTree.tostring(root, encoding="utf-8") + + +def dictize(xmlstr): + root = objectify.fromstring(xmlstr) + return {root.tag: _dictize(root)} + + +def _dictize(e): + d = {} + if e.text is not None: + if not e.attrib and e.countchildren() == 0: + return e.pyval + d['pyval'] = e.pyval + d.update(e.attrib) + for child in e.iterchildren(): + if child.tag in d: + continue + if len(child) > 1: + d[child.tag] = [ + _dictize(same_tag_child) for same_tag_child in child] + else: + d[child.tag] = _dictize(child) + return d diff --git a/tests/test_rest.py b/tests/test_rest.py index 99f2176..f0b828c 100644 --- a/tests/test_rest.py +++ b/tests/test_rest.py @@ -164,9 +164,9 @@ class RestTests(unittest.TestCase): nodedev = json.loads(self.request('/host/devices/scsi_host4').read()) # Mockmodel generates random wwpn and wwnn self.assertEquals('scsi_host4', nodedev['name']) - self.assertEquals('fc_host', nodedev['adapter_type']) - self.assertEquals(16, len(nodedev['wwpn'])) - self.assertEquals(16, len(nodedev['wwnn'])) + self.assertEquals('fc_host', nodedev['adapter']['type']) + self.assertEquals(16, len(nodedev['adapter']['wwpn'])) + self.assertEquals(16, len(nodedev['adapter']['wwnn'])) def test_get_vms(self): vms = json.loads(self.request('/vms').read()) diff --git a/tests/test_storagepool.py b/tests/test_storagepool.py index 2f5e376..3fbeed0 100644 --- a/tests/test_storagepool.py +++ b/tests/test_storagepool.py @@ -145,9 +145,10 @@ class storagepoolTests(unittest.TestCase): 'path': '/dev/disk/by-path', 'source': { 'name': 'scsi_host3', - 'adapter_type': 'fc_host', - 'wwpn': '0123456789abcdef', - 'wwnn': 'abcdef0123456789'}}, + 'adapter': { + 'type': 'fc_host', + 'wwpn': '0123456789abcdef', + 'wwnn': 'abcdef0123456789'}}}, 'xml': """ <pool type='scsi'>

This patch adds a '_passthrough=true' filter to /host/devices, so it can filter and shows all devices eligible to passthrough to guest. Theoretically, all PCI, USB and SCSI devices can be assigned to guest directly. Linux kernel is able to recognize the host IOMMU group layout. If two PCI devices are in the same IOMMU group, it means there are possible interconnections between the devices, and the devices can talk to each other bypassing IOMMU. This implies isolation is not pefect between those devices, so all devices in a IOMMU group must be assigned to guest together. On host that recognizes IOMMU groups, by accessing the URI /host/devices?_passthrough_affected_by=DEVICE_NAME, it returns a list containing the devices in the same IOMMU group as DEVICE_NAME, and all of the children devices of them. So the front-end can show all the affected devices to user, and it helps the user to determine which host devices are to be assigned to guest. How to test: List all types of devices to passthrough curl -k -u root -H "Content-Type: application/json" \ -H "Accept: application/json" \ 'https://127.0.0.1:8001/host/devices?_passthrough=true' List all eligible PCI devices to passthrough /host/devices?_passthrough=true&_cap=pci List all USB devices to passthrough /host/devices?_passthrough=true&_cap=usb_device List all SCSI devices to passthrough /host/devices?_passthrough=true&_cap=scsi List devices in the same IOMMU group as pci_0000_00_19_0 /host/devices?_passthrough_affected_by=pci_0000_00_19_0 v1: v1 series does not contain this patch. v2: Deal with calculation "leaf" device and "affected" device. v5: Change _passthrough=1 to _passthrough=true in the URI scheme. Filter PCI devices according the PCI class. v6: Don't passthrough PCI device of class code 07. In modern x86 machine, it's possible that "6 Series/C200 Series Chipset Family MEI Controller" and "6 Series/C200 Series Chipset Family KT Controller" are of this class code. These two devices are not suitable to passthrough to guest. We don't have simple and reliable way to distinguish normal serial controller and host chipset XXX controller. This type of PCI devices also include various serial, parallel, modem, communication controller. Serial and parallel controllers can be re-direct from ttyS0 to QEMU's pty using socat, and there is little performance benefit to directly assign to guest. So it'k ok not to passththrough PCI device of class code 07. v8: Use a new flag filter "_passthrough_group_by" /host/devices?_passthrough_group_by=pci_XXX instead of using sub-collection /host/devices/pci_XXX/passthrough_affected_devices v9: Use the same LibvirtConnection object as the Model, so as to avoid connection exhausting. v10: Adapt to RHEL 6. RHEL 6 does not provide iommu group information in sysfs. For now we just ignore this error and live with it. The device passthrough for PCI devices will not work, but the basic devices informations are still provided to the user. In future we'll develope code to gather iommu group information. v11: In previous commits, we don't allow to passthrough device with children, and only passthrough the children. It proves it's inflexible and less useful. The PCI class code white list also filter out too much types of devices, and there is no way to cleanly differenciate devices suitable to passthrough using class code. In this patch, we allow Kimchi to passthrough a parent PCI device. The front-end can use the existing "_passthrough_group_by" filter to list the affected children devices and other PCI devices in the same group, as well as the children of those devices. We also drop the class code white list, and filter out only PCI bridge and video cards. libvirt uses domain, bus, slot and function to encode the device name, so sorting the device name results the same effect as sorting based on domain:bus:slot:function. This patch sorts all the devices based on the name. When an SCSI adapter is assigned to virtual machine, the previous node device scsi_host and scsi_target become stale for the host machine. Unfortunately, libvirt only removes the invalid scsi_host device, but not scsi_target. When Kimchi is looking up the parent scsi_host of a scsi_target, the scsi_host device actually does not exist, and it explodes. This patch catches such error and ignores those devices without a valid parent device. v12: Coding style improvements. Use "None" instead of "-1". Signed-off-by: Zhou Zheng Sheng <zhshzhou(a)linux.vnet.ibm.com&gt; --- src/kimchi/i18n.py | 1 + src/kimchi/model/host.py | 33 ++++++++++-- src/kimchi/model/hostdev.py | 121 ++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 147 insertions(+), 8 deletions(-) diff --git a/src/kimchi/i18n.py b/src/kimchi/i18n.py index 1b543ce..98adc46 100644 --- a/src/kimchi/i18n.py +++ b/src/kimchi/i18n.py @@ -233,6 +233,7 @@ messages = { "KCHHOST0001E": _("Unable to shutdown host machine as there are running virtual machines"), "KCHHOST0002E": _("Unable to reboot host machine as there are running virtual machines"), "KCHHOST0003E": _("Node device '%(name)s' not found"), + "KCHHOST0004E": _("Conflicting flag filters specified."), "KCHPKGUPD0001E": _("No packages marked for update"), "KCHPKGUPD0002E": _("Package %(name)s is not marked to be updated."), diff --git a/src/kimchi/model/host.py b/src/kimchi/model/host.py index 7d7cd66..5d31809 100644 --- a/src/kimchi/model/host.py +++ b/src/kimchi/model/host.py @@ -32,8 +32,9 @@ from kimchi import disks from kimchi import netinfo from kimchi import xmlutils from kimchi.basemodel import Singleton -from kimchi.exception import InvalidOperation, NotFoundError, OperationFailed from kimchi.model import hostdev +from kimchi.exception import InvalidOperation, InvalidParameter +from kimchi.exception import NotFoundError, OperationFailed from kimchi.model.config import CapabilitiesModel from kimchi.model.tasks import TaskModel from kimchi.model.vms import DOM_STATE_MAP @@ -299,10 +300,28 @@ class DevicesModel(object): except AttributeError: self.cap_map['fc_host'] = None - def get_list(self, _cap=None): + def get_list(self, _cap=None, _passthrough=None, + _passthrough_affected_by=None): + if _passthrough_affected_by is not None: + # _passthrough_affected_by conflicts with _cap and _passthrough + if (_cap, _passthrough) != (None, None): + raise InvalidParameter("KCHHOST0004E") + return sorted( + self._get_passthrough_affected_devs(_passthrough_affected_by)) + if _cap == 'fc_host': - return self._get_devices_fc_host() - return self._get_devices_with_capability(_cap) + dev_names = self._get_devices_fc_host() + else: + dev_names = self._get_devices_with_capability(_cap) + + if _passthrough is not None and _passthrough.lower() == 'true': + conn = self.conn.get() + passthrough_names = [ + dev['name'] for dev in hostdev.get_passthrough_dev_infos(conn)] + dev_names = list(set(dev_names) & set(passthrough_names)) + + dev_names.sort() + return dev_names def _get_devices_with_capability(self, cap): conn = self.conn.get() @@ -314,6 +333,12 @@ class DevicesModel(object): return [] return [name.name() for name in conn.listAllDevices(cap_flag)] + def _get_passthrough_affected_devs(self, dev_name): + conn = self.conn.get() + info = DeviceModel(conn=self.conn).lookup(dev_name) + affected = hostdev.get_affected_passthrough_devices(conn, info) + return [dev_info['name'] for dev_info in affected] + def _get_devices_fc_host(self): conn = self.conn.get() # Libvirt < 1.0.5 does not support fc_host capability diff --git a/src/kimchi/model/hostdev.py b/src/kimchi/model/hostdev.py index 103c1e7..63cdb21 100644 --- a/src/kimchi/model/hostdev.py +++ b/src/kimchi/model/hostdev.py @@ -17,7 +17,9 @@ # License along with this library; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +import os from pprint import pformat +from pprint import pprint from kimchi.model.libvirtconnection import LibvirtConnection from kimchi.utils import kimchi_log @@ -36,7 +38,13 @@ def _get_dev_info_tree(dev_infos): if dev_info['parent'] is None: root = dev_info continue - parent = devs[dev_info['parent']] + + try: + parent = devs[dev_info['parent']] + except KeyError: + kimchi_log.error('Parent %s of device %s does not exist.', + dev_info['parent'], dev_info['name']) + continue try: children = parent['children'] @@ -47,6 +55,109 @@ def _get_dev_info_tree(dev_infos): return root +def _is_pci_qualified(pci_dev): + # PCI bridge is not suitable to passthrough + # KVM does not support passthrough graphic card now + blacklist_classes = (0x030000, 0x060000) + + with open(os.path.join(pci_dev['path'], 'class')) as f: + pci_class = int(f.readline().strip(), 16) + + if pci_class & 0xff0000 in blacklist_classes: + return False + + return True + + +def get_passthrough_dev_infos(libvirt_conn): + ''' Get devices eligible to be passed through to VM. ''' + + def is_eligible(dev): + return dev['device_type'] in ('usb_device', 'scsi') or \ + (dev['device_type'] == 'pci' and _is_pci_qualified(dev)) + + dev_infos = _get_all_host_dev_infos(libvirt_conn) + + return [dev_info for dev_info in dev_infos if is_eligible(dev_info)] + + +def _get_same_iommugroup_devices(dev_infos, device_info): + dev_dict = dict([(dev_info['name'], dev_info) for dev_info in dev_infos]) + + def get_iommu_group(dev_info): + # Find out the iommu group of a given device. + # Child device belongs to the same iommu group as the parent device. + try: + return dev_info['iommuGroup'] + except KeyError: + pass + + parent = dev_info['parent'] + while parent is not None: + try: + parent_info = dev_dict[parent] + except KeyError: + kimchi_log.error("Parent %s of device %s does not exist", + parent, dev_info['name']) + break + + try: + iommuGroup = parent_info['iommuGroup'] + except KeyError: + pass + else: + return iommuGroup + + parent = parent_info['parent'] + + return None + + iommu_group = get_iommu_group(device_info) + + if iommu_group is None: + return [] + + return [dev_info for dev_info in dev_infos + if dev_info['name'] != device_info['name'] and + get_iommu_group(dev_info) == iommu_group] + + +def _get_children_devices(dev_infos, device_info): + def get_children_recursive(parent): + try: + children = parent['children'] + except KeyError: + return [] + + result = [] + for child in children: + result.append(child) + result.extend(get_children_recursive(child)) + + return result + + # Annotate every the dev_info element with children information + _get_dev_info_tree(dev_infos) + + for dev_info in dev_infos: + if dev_info['name'] == device_info['name']: + return get_children_recursive(dev_info) + + return [] + + +def get_affected_passthrough_devices(libvirt_conn, passthrough_dev): + dev_infos = _get_all_host_dev_infos(libvirt_conn) + + group_devices = _get_same_iommugroup_devices(dev_infos, passthrough_dev) + if not group_devices: + # On host without iommu group support, the affected devices should + # at least include all children devices + group_devices.extend(_get_children_devices(dev_infos, passthrough_dev)) + + return group_devices + + def get_dev_info(node_dev): ''' Parse the node device XML string into dict according to http://libvirt.org/formatnode.html. @@ -168,8 +279,7 @@ def _get_usb_device_dev_info(info): # For test and debug -def _print_host_dev_tree(): - libvirt_conn = LibvirtConnection('qemu:///system').get() +def _print_host_dev_tree(libvirt_conn): dev_infos = _get_all_host_dev_infos(libvirt_conn) root = _get_dev_info_tree(dev_infos) if root is None: @@ -207,4 +317,7 @@ def _format_dev_node(node): if __name__ == '__main__': - _print_host_dev_tree() + libvirt_conn = LibvirtConnection('qemu:///system').get() + _print_host_dev_tree(libvirt_conn) + print 'Eligible passthrough devices:' + pprint(get_passthrough_dev_infos(libvirt_conn))

This patch enbales Kimchi's VM to use host devices directly, and it greatly improves the related device performance. The user can assign PCI, USB and SCSI LUN directly to VM, as long as the host supports one of Intel VT-d, AMD IOMMU or POWER sPAPR technology and runs a recent release of Linux kernel. This patch adds a sub-collection "hostdevs" to the URI vms/vm-name/. The front-end can GET vms/vm-name/hostdevs and vms/vm-name/hostdevs/dev-name or POST (assign) vms/vm-name/hostdevs and DELETE (dismiss) vms/vm-name/hostdevs/dev-name The eligible devices to assign are the devices listed by the URI host/devices?_passthrough=1 When assigning a host PCI device to VM, all the eligible PCI devices in the same IOMMU group are also automatically assigned, and vice versa when dismissing a host PIC device from the VM. Some examples: Assign a USB device: curl -k -u root -H "Content-Type: application/json" \ -H "Accept: application/json" \ -X POST -d '{"name": "usb_1_1_6"}' \ 'https://127.0.0.1:8001/vms/rhel65/hostdevs' Assign a PCI device: -d '{"name": "pci_0000_0d_00_0"}' Assign a SCSI LUN: -d '{"name": "scsi_1_0_0_0"}' List assigned devices: curl -k -u root -H "Content-Type: application/json" \ -H "Accept: application/json" \ 'https://127.0.0.1:8001/vms/rhel65/hostdevs' The above command should print following. [ { "type":"scsi", "name":"scsi_1_0_0_0" }, { "type":"usb", "name":"usb_1_1_6" }, { "type":"pci", "name":"pci_0000_0d_00_0" }, { "type":"pci", "name":"pci_0000_03_00_0" } ] Notice that the device pci_0000_03_00_0 is also assigned automatically. The assigned devices are hot-plugged to VM and also written to the domain XML. When it's possible, it enables VFIO for PCI device assignment. On distribution with old Linux kernel, there are many limitations with PCI passthrough and it's hardly useful. This patch tries to adapt to old kernel but it's better to use a newer kernel with vfio support. Thus this patch also provide a new capability in /config/capabilities. The front-end can disable or freeze the related web UI if back-end reports host does not support vfio. curl -k -u root -H "Content-Type: application/json" \ -H "Accept: application/json" \ 'https://127.0.0.1:8001/config/capabilities' The above command should print following. { "blah": "blah", ... "kernel_vfio":true } v1: Handle the devices in the VM template. v2: Handle the devices in the VM sub-resource "hostdevs". v3: No change. v4: Not all domain XMLs contain hostdev node. Deal with the case. v5: Change _passthrough='1' to _passthrough='true'. When attaching and detaching a device, do not use VIR_DOMAIN_AFFECT_CURRENT flag, instead, use kimchi.model.utils.get_vm_config_flag() to correctly set the device flag. v11: Add Capability kernel_vfio to indicate if Linux kernel is new enough to support vfio. v12: Use lxml.etree and lxml.builder to generate XML. Signed-off-by: Zhou Zheng Sheng <zhshzhou(a)linux.vnet.ibm.com&gt; --- src/kimchi/control/vm/hostdevs.py | 43 ++++++ src/kimchi/featuretests.py | 10 +- src/kimchi/i18n.py | 7 + src/kimchi/model/config.py | 6 +- src/kimchi/model/vmhostdevs.py | 295 ++++++++++++++++++++++++++++++++++++++ src/kimchi/rollbackcontext.py | 3 + 6 files changed, 362 insertions(+), 2 deletions(-) create mode 100644 src/kimchi/control/vm/hostdevs.py create mode 100644 src/kimchi/model/vmhostdevs.py diff --git a/src/kimchi/control/vm/hostdevs.py b/src/kimchi/control/vm/hostdevs.py new file mode 100644 index 0000000..1eb88cf --- /dev/null +++ b/src/kimchi/control/vm/hostdevs.py @@ -0,0 +1,43 @@ +# +# Project Kimchi +# +# Copyright IBM, Corp. 2014 +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +from kimchi.control.base import Collection, Resource +from kimchi.control.utils import UrlSubNode + + +@UrlSubNode("hostdevs") +class VMHostDevs(Collection): + def __init__(self, model, vmid): + super(VMHostDevs, self).__init__(model) + self.resource = VMHostDev + self.vmid = vmid + self.resource_args = [self.vmid, ] + self.model_args = [self.vmid, ] + + +class VMHostDev(Resource): + def __init__(self, model, vmid, ident): + super(VMHostDev, self).__init__(model, ident) + self.vmid = vmid + self.ident = ident + self.model_args = [self.vmid, self.ident] + + @property + def data(self): + return self.info diff --git a/src/kimchi/featuretests.py b/src/kimchi/featuretests.py index 8964098..c1470fc 100644 --- a/src/kimchi/featuretests.py +++ b/src/kimchi/featuretests.py @@ -29,7 +29,7 @@ from lxml.builder import E from kimchi.rollbackcontext import RollbackContext -from kimchi.utils import kimchi_log +from kimchi.utils import kimchi_log, run_command ISO_STREAM_XML = """ @@ -206,3 +206,11 @@ class FeatureTests(object): return True except libvirt.libvirtError: return False + + @staticmethod + def kernel_support_vfio(): + out, err, rc = run_command(['modprobe', 'vfio-pci']) + if rc != 0: + kimchi_log.warning("Unable to load Kernal module vfio-pci.") + return False + return True diff --git a/src/kimchi/i18n.py b/src/kimchi/i18n.py index 98adc46..ad65775 100644 --- a/src/kimchi/i18n.py +++ b/src/kimchi/i18n.py @@ -98,6 +98,13 @@ messages = { "KCHVM0031E": _("The guest console password must be a string."), "KCHVM0032E": _("The life time for the guest console password must be a number."), + "KCHVMHDEV0001E": _("VM %(vmid)s does not contain directly assigned host device %(dev_name)s."), + "KCHVMHDEV0002E": _("The host device %(dev_name)s is not allowed to directly assign to VM."), + "KCHVMHDEV0003E": _("No IOMMU groups found. Host PCI pass through needs IOMMU group to function correctly. " + "Please enable Intel VT-d or AMD IOMMU in your BIOS, then verify the Kernel is compiled with IOMMU support. " + "For Intel CPU, add intel_iommu=on to your Kernel parameter in /boot/grub2/grub.conf. " + "For AMD CPU, add iommu=pt iommu=1."), + "KCHVMIF0001E": _("Interface %(iface)s does not exist in virtual machine %(name)s"), "KCHVMIF0002E": _("Network %(network)s specified for virtual machine %(name)s does not exist"), "KCHVMIF0003E": _("Do not support guest interface hot plug attachment"), diff --git a/src/kimchi/model/config.py b/src/kimchi/model/config.py index 1c00cfe..9ffc53e 100644 --- a/src/kimchi/model/config.py +++ b/src/kimchi/model/config.py @@ -53,6 +53,7 @@ class CapabilitiesModel(object): self.libvirt_stream_protocols = [] self.fc_host_support = False self.metadata_support = False + self.kernel_vfio = False # Subscribe function to set host capabilities to be run when cherrypy # server is up @@ -66,6 +67,7 @@ class CapabilitiesModel(object): self.nfs_target_probe = FeatureTests.libvirt_support_nfs_probe() self.fc_host_support = FeatureTests.libvirt_support_fc_host() self.metadata_support = FeatureTests.has_metadata_support() + self.kernel_vfio = FeatureTests.kernel_support_vfio() self.libvirt_stream_protocols = [] for p in ['http', 'https', 'ftp', 'ftps', 'tftp']: @@ -110,7 +112,9 @@ class CapabilitiesModel(object): 'system_report_tool': bool(report_tool), 'update_tool': update_tool, 'repo_mngt_tool': repo_mngt_tool, - 'federation': kconfig.get("server", "federation")} + 'federation': kconfig.get("server", "federation"), + 'kernel_vfio': self.kernel_vfio, + } class DistrosModel(object): diff --git a/src/kimchi/model/vmhostdevs.py b/src/kimchi/model/vmhostdevs.py new file mode 100644 index 0000000..d9e7a05 --- /dev/null +++ b/src/kimchi/model/vmhostdevs.py @@ -0,0 +1,295 @@ +# +# Project Kimchi +# +# Copyright IBM, Corp. 2014 +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +import glob +import os + +import libvirt +from lxml import etree, objectify +from lxml.builder import E + +from kimchi.exception import InvalidOperation, InvalidParameter, NotFoundError +from kimchi.model.config import CapabilitiesModel +from kimchi.model.host import DeviceModel, DevicesModel +from kimchi.model.utils import get_vm_config_flag +from kimchi.model.vms import DOM_STATE_MAP, VMModel +from kimchi.rollbackcontext import RollbackContext +from kimchi.utils import kimchi_log, run_command + + +class VMHostDevsModel(object): + def __init__(self, **kargs): + self.conn = kargs['conn'] + + def get_list(self, vmid): + dom = VMModel.get_vm(vmid, self.conn) + xmlstr = dom.XMLDesc(0) + root = objectify.fromstring(xmlstr) + try: + hostdev = root.devices.hostdev + except AttributeError: + return [] + + return [self._deduce_dev_name(e) for e in hostdev] + + @staticmethod + def _toint(num_str): + if num_str.startswith('0x'): + return int(num_str, 16) + elif num_str.startswith('0'): + return int(num_str, 8) + else: + return int(num_str) + + def _deduce_dev_name(self, e): + return getattr(self, '_deduce_dev_name_%s' % e.attrib['type'])(e) + + def _deduce_dev_name_pci(self, e): + attrib = {} + for field in ('domain', 'bus', 'slot', 'function'): + attrib[field] = self._toint(e.source.address.attrib[field]) + return 'pci_%(domain)04x_%(bus)02x_%(slot)02x_%(function)x' % attrib + + def _deduce_dev_name_scsi(self, e): + attrib = {} + for field in ('bus', 'target', 'unit'): + attrib[field] = self._toint(e.source.address.attrib[field]) + attrib['host'] = self._toint( + e.source.adapter.attrib['name'][len('scsi_host'):]) + return 'scsi_%(host)d_%(bus)d_%(target)d_%(unit)d' % attrib + + def _deduce_dev_name_usb(self, e): + dev_names = DevicesModel(conn=self.conn).get_list(_cap='usb_device') + usb_infos = [DeviceModel(conn=self.conn).lookup(dev_name) + for dev_name in dev_names] + + unknown_dev = None + + try: + evendor = self._toint(e.source.vendor.attrib['id']) + eproduct = self._toint(e.source.product.attrib['id']) + except AttributeError: + evendor = 0 + eproduct = 0 + else: + unknown_dev = 'usb_vendor_%s_product_%s' % (evendor, eproduct) + + try: + ebus = self._toint(e.source.address.attrib['bus']) + edevice = self._toint(e.source.address.attrib['device']) + except AttributeError: + ebus = -1 + edevice = -1 + else: + unknown_dev = 'usb_bus_%s_device_%s' % (ebus, edevice) + + for usb_info in usb_infos: + ivendor = self._toint(usb_info['vendor']['id']) + iproduct = self._toint(usb_info['product']['id']) + if evendor == ivendor and eproduct == iproduct: + return usb_info['name'] + ibus = usb_info['bus'] + idevice = usb_info['device'] + if ebus == ibus and edevice == idevice: + return usb_info['name'] + return unknown_dev + + def _passthrough_device_validate(self, dev_name): + eligible_dev_names = \ + DevicesModel(conn=self.conn).get_list(_passthrough='true') + if dev_name not in eligible_dev_names: + raise InvalidParameter('KCHVMHDEV0002E', {'dev_name': dev_name}) + + def create(self, vmid, params): + dev_name = params['name'] + self._passthrough_device_validate(dev_name) + dev_info = DeviceModel(conn=self.conn).lookup(dev_name) + attach_device = getattr( + self, '_attach_%s_device' % dev_info['device_type']) + return attach_device(vmid, dev_info) + + def _get_pci_device_xml(self, dev_info): + if 'detach_driver' not in dev_info: + dev_info['detach_driver'] = 'kvm' + + source = E.source(E.address(domain=str(dev_info['domain']), + bus=str(dev_info['bus']), + slot=str(dev_info['slot']), + function=str(dev_info['function']))) + driver = E.driver(name=dev_info['detach_driver']) + host_dev = E.hostdev(source, driver, + mode='subsystem', type='pci', managed='yes') + + return etree.tostring(host_dev) + + @staticmethod + def _validate_pci_passthrough_env(): + # Linux kernel < 3.5 doesn't provide /sys/kernel/iommu_groups + if os.path.isdir('/sys/kernel/iommu_groups'): + if not glob.glob('/sys/kernel/iommu_groups/*'): + raise InvalidOperation("KCHVMHDEV0003E") + + # Enable virt_use_sysfs on RHEL6 and older distributions + # In recent Fedora, there is no virt_use_sysfs. + out, err, rc = run_command(['getsebool', 'virt_use_sysfs']) + if rc == 0 and out.rstrip('\n') != "virt_use_sysfs --> on": + out, err, rc = run_command(['setsebool', '-P', + 'virt_use_sysfs=on']) + if rc != 0: + kimchi_log.warning("Unable to turn on sebool virt_use_sysfs") + + def _attach_pci_device(self, vmid, dev_info): + self._validate_pci_passthrough_env() + + dom = VMModel.get_vm(vmid, self.conn) + # Due to libvirt limitation, we don't support live assigne device to + # vfio driver. + driver = ('vfio' if DOM_STATE_MAP[dom.info()[0]] == "shutoff" and + CapabilitiesModel().kernel_vfio else 'kvm') + + # Attach all PCI devices in the same IOMMU group + dev_model = DeviceModel(conn=self.conn) + devs_model = DevicesModel(conn=self.conn) + affected_names = devs_model.get_list( + _passthrough_affected_by=dev_info['name']) + passthrough_names = devs_model.get_list( + _cap='pci', _passthrough='true') + group_names = list(set(affected_names) & set(passthrough_names)) + pci_infos = [dev_model.lookup(dev_name) for dev_name in group_names] + pci_infos.append(dev_info) + + device_flags = get_vm_config_flag(dom, mode='all') + + with RollbackContext() as rollback: + for pci_info in pci_infos: + pci_info['detach_driver'] = driver + xmlstr = self._get_pci_device_xml(pci_info) + try: + dom.attachDeviceFlags(xmlstr, device_flags) + except libvirt.libvirtError: + kimchi_log.error( + 'Failed to attach host device %s to VM %s: \n%s', + pci_info['name'], vmid, xmlstr) + raise + rollback.prependDefer(dom.detachDeviceFlags, + xmlstr, device_flags) + rollback.commitAll() + + return dev_info['name'] + + def _get_scsi_device_xml(self, dev_info): + adapter = E.adapter(name=('scsi_host%s' % dev_info['host'])) + address = E.address(type='scsi', bus=str(dev_info['bus']), + target=str(dev_info['target']), + unit=str(dev_info['lun'])) + host_dev = E.hostdev(E.source(adapter, address), + mode='subsystem', type='scsi', sgio='unfiltered') + return etree.tostring(host_dev) + + def _attach_scsi_device(self, vmid, dev_info): + xmlstr = self._get_scsi_device_xml(dev_info) + dom = VMModel.get_vm(vmid, self.conn) + dom.attachDeviceFlags(xmlstr, get_vm_config_flag(dom, mode='all')) + return dev_info['name'] + + def _get_usb_device_xml(self, dev_info): + source = E.source( + E.vendor(id=dev_info['vendor']['id']), + E.product(id=dev_info['product']['id']), + E.address(bus=str(dev_info['bus']), + device=str(dev_info['device'])), + startupPolicy='optional') + host_dev = E.hostdev(source, mode='subsystem', + ype='usb', managed='yes') + return etree.tostring(host_dev) + + def _attach_usb_device(self, vmid, dev_info): + xmlstr = self._get_usb_device_xml(dev_info) + dom = VMModel.get_vm(vmid, self.conn) + dom.attachDeviceFlags(xmlstr, get_vm_config_flag(dom, mode='all')) + return dev_info['name'] + + +class VMHostDevModel(object): + def __init__(self, **kargs): + self.conn = kargs['conn'] + + def lookup(self, vmid, dev_name): + dom = VMModel.get_vm(vmid, self.conn) + xmlstr = dom.XMLDesc(0) + root = objectify.fromstring(xmlstr) + try: + hostdev = root.devices.hostdev + except AttributeError: + raise NotFoundError('KCHVMHDEV0001E', + {'vmid': vmid, 'dev_name': dev_name}) + + devsmodel = VMHostDevsModel(conn=self.conn) + + for e in hostdev: + deduced_name = devsmodel._deduce_dev_name(e) + if deduced_name == dev_name: + return {'name': dev_name, 'type': e.attrib['type']} + + raise NotFoundError('KCHVMHDEV0001E', + {'vmid': vmid, 'dev_name': dev_name}) + + def delete(self, vmid, dev_name): + dom = VMModel.get_vm(vmid, self.conn) + xmlstr = dom.XMLDesc(0) + root = objectify.fromstring(xmlstr) + + try: + hostdev = root.devices.hostdev + except AttributeError: + raise NotFoundError('KCHVMHDEV0001E', + {'vmid': vmid, 'dev_name': dev_name}) + + devsmodel = VMHostDevsModel(conn=self.conn) + pci_devs = [(devsmodel._deduce_dev_name(e), e) for e in hostdev + if e.attrib['type'] == 'pci'] + + for e in hostdev: + if devsmodel._deduce_dev_name(e) == dev_name: + xmlstr = etree.tostring(e) + dom.detachDeviceFlags( + xmlstr, get_vm_config_flag(dom, mode='all')) + if e.attrib['type'] == 'pci': + self._delete_affected_pci_devices(dom, dev_name, pci_devs) + break + else: + raise NotFoundError('KCHVMHDEV0001E', + {'vmid': vmid, 'dev_name': dev_name}) + + def _delete_affected_pci_devices(self, dom, dev_name, pci_devs): + dev_model = DeviceModel(conn=self.conn) + try: + dev_model.lookup(dev_name) + except NotFoundError: + return + + affected_names = set( + DevicesModel( + conn=self.conn).get_list(_passthrough_affected_by=dev_name)) + + for pci_name, e in pci_devs: + if pci_name in affected_names: + xmlstr = etree.tostring(e) + dom.detachDeviceFlags( + xmlstr, get_vm_config_flag(dom, mode='all')) diff --git a/src/kimchi/rollbackcontext.py b/src/kimchi/rollbackcontext.py index 29c0235..2f3e8bc 100644 --- a/src/kimchi/rollbackcontext.py +++ b/src/kimchi/rollbackcontext.py @@ -64,3 +64,6 @@ class RollbackContext(object): def prependDefer(self, func, *args, **kwargs): self._finally.insert(0, (func, args, kwargs)) + + def commitAll(self): + self._finally = []

Add a "vm_holders" sub-collection under host device resource, so the front-end can determine if a device is busy or not, and the user can know which VMs are holding the device. This patch scans all VM XML to check if a device is hold by a VM. Example curl -k -u root -H "Content-Type: application/json" \ -H "Accept: application/json" \ 'https://127.0.0.1:8001/host/devices/usb_1_1_6/vm_holders' Should output a list like following. [ { "state":"shutoff", "name":"fedora20" }, { "state":"running", "name":"f20xfce-slave" } ] If there is no VM holding the device, it prints an empty list []. v5: When assigning a device to VM, check if there are other VMs holding the device and raise an exception. Move the VMHoldersModel to vmhostdevs.py to avoid circular import problem. v11: Allow Guests to Share a Host PCI Device. A PCI device can be shared by guests, as long as there is only one guest active. Signed-off-by: Zhou Zheng Sheng <zhshzhou(a)linux.vnet.ibm.com&gt; --- src/kimchi/control/host.py | 7 +++++++ src/kimchi/model/vmhostdevs.py | 19 +++++++++++++++++++ 2 files changed, 26 insertions(+) diff --git a/src/kimchi/control/host.py b/src/kimchi/control/host.py index 1eb6350..7bcae72 100644 --- a/src/kimchi/control/host.py +++ b/src/kimchi/control/host.py @@ -113,11 +113,18 @@ class Devices(Collection): self.resource = Device +class VMHolders(SimpleCollection): + def __init__(self, model, device_id): + super(VMHolders, self).__init__(model) + self.model_args = (device_id, ) + + class Device(Resource): def __init__(self, model, id): self.role_key = 'storage' self.admin_methods = ['GET'] super(Device, self).__init__(model, id) + self.vm_holders = VMHolders(self.model, id) @property def data(self): diff --git a/src/kimchi/model/vmhostdevs.py b/src/kimchi/model/vmhostdevs.py index d9e7a05..1098f88 100644 --- a/src/kimchi/model/vmhostdevs.py +++ b/src/kimchi/model/vmhostdevs.py @@ -293,3 +293,22 @@ class VMHostDevModel(object): xmlstr = etree.tostring(e) dom.detachDeviceFlags( xmlstr, get_vm_config_flag(dom, mode='all')) + + +class VMHoldersModel(object): + def __init__(self, **kargs): + self.conn = kargs['conn'] + + def get_list(self, device_id): + devsmodel = VMHostDevsModel(conn=self.conn) + + conn = self.conn.get() + doms = conn.listAllDomains(0) + + res = [] + for dom in doms: + dom_name = dom.name() + if device_id in devsmodel.get_list(dom_name): + state = DOM_STATE_MAP[dom.info()[0]] + res.append({"name": dom_name, "state": state}) + return res

Add some basice unit tests for fetching host device information. Update MockModel as well. Update "API.json" and "API.md" to reflect the change in API. v12: Add "error" field for each new parameter and API call in API.json. Define Mock device information directly instead of parsing fake XML. Signed-off-by: Zhou Zheng Sheng <zhshzhou(a)linux.vnet.ibm.com&gt; --- docs/API.md | 57 ++++++++++++++++++++++++++--- src/kimchi/API.json | 38 ++++++++++++++++++++ src/kimchi/i18n.py | 5 +++ src/kimchi/mockmodel.py | 95 ++++++++++++++++++++++++++++++++++++++++++++----- tests/test_model.py | 31 ++++++++++++++++ tests/test_rest.py | 6 ++-- 6 files changed, 215 insertions(+), 17 deletions(-) diff --git a/docs/API.md b/docs/API.md index b65f211..92fbbd5 100644 --- a/docs/API.md +++ b/docs/API.md @@ -168,6 +168,21 @@ Represents a snapshot of the Virtual Machine's primary monitor. **Actions (POST):** +### Sub-collection: Virtual Machine Passthrough Devices +**URI:** /vms/*:name*/hostdevs +* **GET**: Retrieve a summarized list of all directly assigned host device of + specified guest. +* **POST**: Directly assign a host device to guest. + * name: The name of the host device to be assigned to vm. + +### Sub-resource: Device +**URI:** /vms/*:name*/hostdevs/*:dev* +* **GET**: Retrieve assigned device information + * name: The name of the assigned device. + * type: The type of the assigned device. +* **DELETE**: Detach the host device from VM. + + ### Collection: Templates **URI:** /templates @@ -897,11 +912,17 @@ stats history **Methods:** -* **GET**: Retrieves list of host pci devices (Node Devices). - Currently only scsi_host devices are supported: +* **GET**: Retrieves list of host devices (Node Devices). * Parameters: * _cap: Filter node device list with given node device capability. To list Fibre Channel SCSI Host devices, use "_cap=fc_host". + Other available values are "fc_host", "net", "pci", "scsi", + "storage", "system", "usb" and "usb_device". + * _passthrough: Filter devices eligible to be assigned to guest + directly. Possible values are "ture" and "false". + * _passthrough_affected_by: Filter the affected devices in the same + group of a certain directly assigned device. + The value should be the name of a device. ### Resource: Device @@ -909,14 +930,40 @@ stats history **Methods:** -* **GET**: Retrieve information of a single pci device. - Currently only scsi_host devices are supported: +* **GET**: Retrieve information of a single host device. + * device_type: Type of the device, supported types are "net", "pci", "scsi", + "storage", "system", "usb" and "usb_device". * name: The name of the device. * path: Path of device in sysfs. - * adapter: Host adapter information. Empty if pci device is not scsi_host. + * parent: The name of the parent parent device. + * adapter: Host adapter information of a "scsi_host" or "fc_host" device. * type: The capability type of the scsi_host device (fc_host, vport_ops). * wwnn: The HBA Word Wide Node Name. Empty if pci device is not fc_host. * wwpn: The HBA Word Wide Port Name. Empty if pci device is not fc_host. + * domain: Domain number of a "pci" device. + * bus: Bus number of a "pci" device. + * slot: Slot number of a "pci" device. + * function: Function number of a "pci" device. + * vendor: Vendor information of a "pci" device. + * id: Vendor id of a "pci" device. + * description: Vendor description of a "pci" device. + * product: Product information of a "pci" device. + * id: Product id of a "pci" device. + * description: Product description of a "pci" device. + * iommuGroup: IOMMU group number of a "pci" device. Would be None/null if + host does not enable IOMMU support. + + +### Sub-collection: VMs with the device assigned. +**URI:** /host/devices/*:name*/vmholders +* **GET**: Retrieve a summarized list of all VMs holding the device. + +### Sub-resource: VM holder +**URI:** /host/devices/*:name*/vmholders/*:vm* +* **GET**: Retrieve information of the VM which is holding the device + * name: The name of the VM. + * state: The power state of the VM. Could be "running" and "shutdown". + ### Collection: Host Packages Update diff --git a/src/kimchi/API.json b/src/kimchi/API.json index 5b752dc..d9e13f0 100644 --- a/src/kimchi/API.json +++ b/src/kimchi/API.json @@ -721,6 +721,44 @@ }, "additionalProperties": false, "error": "KCHAPI0001E" + }, + "devices_get_list": { + "type": "object", + "properties": { + "_cap": { + "description": "List specific type of device", + "type": "string", + "pattern": "^fc_host|net|pci|scsi|scsi_host|storage|system|usb|usb_device$", + "error": "KCHDEVS0001E" + }, + "_passthrough": { + "description": "List only devices eligible to be assigned to guest", + "type": "string", + "pattern": "^true|false$", + "error": "KCHDEVS0002E" + }, + "_passthrough_affected_by": { + "description": "List the affected devices in the same group of a certain device to be assigned to guest", + "type": "string", + "pattern": "^[_A-Za-z0-9-]+$", + "error": "KCHDEVS0003E" + } + }, + "additionalProperties": false, + "error": "KCHAPI0001E" + }, + "vmhostdevs_create": { + "type": "object", + "properties": { + "name": { + "description": "Then name of the device to assign to VM", + "type": "string", + "pattern": "^[_A-Za-z0-9-]+$", + "required": true, + "error": "KCHVMHDEV0004E" + } + }, + "error": "KCHAPI0001E" } } } diff --git a/src/kimchi/i18n.py b/src/kimchi/i18n.py index ad65775..75fb076 100644 --- a/src/kimchi/i18n.py +++ b/src/kimchi/i18n.py @@ -40,6 +40,10 @@ messages = { "KCHAUTH0002E": _("You are not authorized to access Kimchi"), "KCHAUTH0003E": _("Specify %(item)s to login into Kimchi"), + "KCHDEVS0001E": _('Unknown "_cap" specified'), + "KCHDEVS0002E": _('"_passthrough" should be "true" or "false"'), + "KCHDEVS0003E": _('"_passthrough_affected_by" should be a device name string'), + "KCHDISKS0001E": _("Error while getting block devices. Details: %(err)s"), "KCHDISKS0002E": _("Error while getting block device information for %(device)s."), @@ -104,6 +108,7 @@ messages = { "Please enable Intel VT-d or AMD IOMMU in your BIOS, then verify the Kernel is compiled with IOMMU support. " "For Intel CPU, add intel_iommu=on to your Kernel parameter in /boot/grub2/grub.conf. " "For AMD CPU, add iommu=pt iommu=1."), + "KCHVMHDEV0004E": _('"name" should be a device name string'), "KCHVMIF0001E": _("Interface %(iface)s does not exist in virtual machine %(name)s"), "KCHVMIF0002E": _("Network %(network)s specified for virtual machine %(name)s does not exist"), diff --git a/src/kimchi/mockmodel.py b/src/kimchi/mockmodel.py index 0fa16e8..2d0135a 100644 --- a/src/kimchi/mockmodel.py +++ b/src/kimchi/mockmodel.py @@ -91,6 +91,7 @@ class MockModel(object): self.next_taskid = 1 self.storagepool_activate('default') self._mock_host_repositories = MockRepositories() + self._mock_devices = MockDevices() def _static_vm_update(self, dom, params): state = dom.info['state'] @@ -611,16 +612,15 @@ class MockModel(object): raise InvalidOperation("KCHVOL0006E", {'pool': pool}) return res._volumes.keys() - def devices_get_list(self, _cap=None): - return ['scsi_host3', 'scsi_host4', 'scsi_host5'] + def devices_get_list(self, _cap=None, _passthrough=None, + _passthrough_affected_by=None): + if _cap is None: + return self._mock_devices.devices.keys() + return [dev['name'] for dev in self._mock_devices.devices.values() + if dev['device_type'] == _cap] - def device_lookup(self, nodedev_name): - return { - 'name': nodedev_name, - 'adapter': { - 'type': 'fc_host', - 'wwnn': uuid.uuid4().hex[:16], - 'wwpn': uuid.uuid4().hex[:16]}} + def device_lookup(self, dev_name): + return self._mock_devices.devices[dev_name] def isopool_lookup(self, name): return {'state': 'active', @@ -1418,6 +1418,83 @@ class MockRepositories(object): del self._repos[repo_id] +class MockDevices(object): + def __init__(self): + self.devices = { + 'computer': {'device_type': 'system', + 'firmware': {'release_date': '01/01/2012', + 'vendor': 'LENOVO', + 'version': 'XXXXX (X.XX )'}, + 'hardware': {'serial': 'PXXXXX', + 'uuid': + '9d660370-820f-4241-8731-5a60c97e8aa6', + 'vendor': 'LENOVO', + 'version': 'ThinkPad T420'}, + 'name': 'computer', + 'parent': None, + 'product': '4180XXX'}, + 'pci_0000_03_00_0': {'bus': 3, + 'device_type': 'pci', + 'domain': 0, + 'driver': {'name': 'iwlwifi'}, + 'function': 0, + 'iommuGroup': 7, + 'name': 'pci_0000_03_00_0', + 'parent': 'computer', + 'path': + '/sys/devices/pci0000:00/0000:03:00.0', + 'product': { + 'description': + 'Centrino Advanced-N 6205 [Taylor Peak]', + 'id': '0x0085'}, + 'slot': 0, + 'vendor': {'description': 'Intel Corporation', + 'id': '0x8086'}}, + 'pci_0000_0d_00_0': {'bus': 13, + 'device_type': 'pci', + 'domain': 0, + 'driver': {'name': 'sdhci-pci'}, + 'function': 0, + 'iommuGroup': 7, + 'name': 'pci_0000_0d_00_0', + 'parent': 'computer', + 'path': + '/sys/devices/pci0000:00/0000:0d:00.0', + 'product': {'description': + 'PCIe SDXC/MMC Host Controller', + 'id': '0xe823'}, + 'slot': 0, + 'vendor': {'description': 'Ricoh Co Ltd', + 'id': '0x1180'}}, + 'scsi_host0': {'adapter': {'fabric_wwn': '37df6c1efa1b4388', + 'type': 'fc_host', + 'wwnn': 'efb6563f06434a98', + 'wwpn': '742f32073aab45d7'}, + 'device_type': 'scsi_host', + 'host': 0, + 'name': 'scsi_host0', + 'parent': 'computer', + 'path': '/sys/devices/pci0000:00/0000:40:00.0/0'}, + 'scsi_host1': {'adapter': {'fabric_wwn': '542efa5dced34123', + 'type': 'fc_host', + 'wwnn': 'b7433a40c9b84092', + 'wwpn': '25c1f485ae42497f'}, + 'device_type': 'scsi_host', + 'host': 0, + 'name': 'scsi_host1', + 'parent': 'computer', + 'path': '/sys/devices/pci0000:00/0000:40:00.0/1'}, + 'scsi_host2': {'adapter': {'fabric_wwn': '5c373c334c20478d', + 'type': 'fc_host', + 'wwnn': 'f2030bec4a254e6b', + 'wwpn': '07dbca4164d44096'}, + 'device_type': 'scsi_host', + 'host': 0, + 'name': 'scsi_host2', + 'parent': 'computer', + 'path': '/sys/devices/pci0000:00/0000:40:00.0/2'}} + + def get_mock_environment(): model = MockModel() for i in xrange(5): diff --git a/tests/test_model.py b/tests/test_model.py index 1f2e79c..f0f5fd8 100644 --- a/tests/test_model.py +++ b/tests/test_model.py @@ -1059,6 +1059,37 @@ class ModelTests(unittest.TestCase): self.assertIn('ipaddr', iface) self.assertIn('netmask', iface) + @unittest.skipUnless(utils.running_as_root(), 'Must be run as root') + def test_get_devices(self): + def asset_devices_type(devices, dev_type): + for dev in devices: + self.assertEquals(dev['device_type'], dev_type) + + inst = model.Model('qemu:///system', + objstore_loc=self.tmp_store) + + devs = inst.devices_get_list() + + for dev_type in ('pci', 'usb_device', 'scsi'): + names = inst.devices_get_list(_cap=dev_type) + self.assertTrue(set(names) <= set(devs)) + infos = [inst.device_lookup(name) for name in names] + asset_devices_type(infos, dev_type) + + passthru_devs = inst.devices_get_list(_passthrough='true') + self.assertTrue(set(passthru_devs) <= set(devs)) + + for dev_type in ('pci', 'usb_device', 'scsi'): + names = inst.devices_get_list(_cap=dev_type, _passthrough='true') + self.assertTrue(set(names) <= set(devs)) + infos = [inst.device_lookup(name) for name in names] + asset_devices_type(infos, dev_type) + + for dev_name in passthru_devs: + affected_devs = inst.devices_get_list( + _passthrough_affected_by=dev_name) + self.assertTrue(set(affected_devs) <= set(devs)) + def test_async_tasks(self): class task_except(Exception): pass diff --git a/tests/test_rest.py b/tests/test_rest.py index f0b828c..a1ebb32 100644 --- a/tests/test_rest.py +++ b/tests/test_rest.py @@ -157,13 +157,13 @@ class RestTests(unittest.TestCase): self.assertHTTPStatus(406, "/", None, 'GET', h) def test_host_devices(self): - nodedevs = json.loads(self.request('/host/devices').read()) + nodedevs = json.loads(self.request('/host/devices?_cap=scsi_host').read()) # Mockmodel brings 3 preconfigured scsi fc_host self.assertEquals(3, len(nodedevs)) - nodedev = json.loads(self.request('/host/devices/scsi_host4').read()) + nodedev = json.loads(self.request('/host/devices/scsi_host2').read()) # Mockmodel generates random wwpn and wwnn - self.assertEquals('scsi_host4', nodedev['name']) + self.assertEquals('scsi_host2', nodedev['name']) self.assertEquals('fc_host', nodedev['adapter']['type']) self.assertEquals(16, len(nodedev['adapter']['wwpn'])) self.assertEquals(16, len(nodedev['adapter']['wwnn']))

From: Yu Xin Huo <huoyuxin(a)linux.vnet.ibm.com&gt; Signed-off-by: Yu Xin Huo <huoyuxin(a)linux.vnet.ibm.com&gt; Signed-off-by: Wen Wang <wenwang(a)linux.vnet.ibm.com&gt; --- ui/css/theme-default/guest-edit.css | 86 ++++++++++++++++++++++++++++++++++++- ui/js/src/kimchi.api.js | 55 ++++++++++++++++++++++++ ui/js/src/kimchi.guest_edit_main.js | 81 ++++++++++++++++++++++++++++++++++ ui/pages/guest-edit.html.tmpl | 28 ++++++++++++ 4 files changed, 248 insertions(+), 2 deletions(-) diff --git a/ui/css/theme-default/guest-edit.css b/ui/css/theme-default/guest-edit.css index 76fbaf2..ef2266f 100644 --- a/ui/css/theme-default/guest-edit.css +++ b/ui/css/theme-default/guest-edit.css @@ -17,8 +17,8 @@ */ #guest-edit-window { font-size: 13px; - height: 400px; - width: 610px; + height: 420px; + width: 820px; } #guest-edit-tabs { @@ -261,3 +261,85 @@ width: 46%; float: right; } + +.guest-edit-pci { + height: 79%; + overflow: auto; + font-size: 12px; +} + +.guest-edit-pci .filter { + height: 35px; + margin-right: 5px; + overflow: hidden; +} + +.guest-edit-pci .group { + float: right; +} + +.guest-edit-pci .filter .control { + border: 1px solid #AAAAAA; + font-size: 12px; + background-color: white; +} + +.guest-edit-pci .filter select { + border-right: 0px!important; + border-radius: 7px 0px 0px 7px; + padding: 2px 2px 2px 7px; + width: 100px; +} + +.guest-edit-pci .filter select option { + padding-left: 7px; +} + +.guest-edit-pci .filter input { + border-radius: 0px 7px 7px 0px; + padding: 3px 3px 3px 10px; + width: 200px; + font-style: italic; +} + +.guest-edit-pci .header { + margin-bottom: 8px; + padding-bottom: 2px; + font-weight: bold; + border-bottom: 1px solid #999999; +} + +.guest-edit-pci .item { + margin-bottom: 4px; + overflow: hidden; +} + +.guest-edit-pci .cell { + display: inline-block; + vertical-align: middle; + margin-right: 10px; + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + +.guest-edit-pci .item button { + width: 20px; + height: 20px; + float: right; +} + +.guest-edit-pci .name { + width: 18%; + max-width: 18%; +} + +.guest-edit-pci .product { + width: 45%; + max-width: 45%; +} + +.guest-edit-pci .vendor { + width: 25%; + max-width: 25%; +} diff --git a/ui/js/src/kimchi.api.js b/ui/js/src/kimchi.api.js index 3398bd4..86bfd22 100644 --- a/ui/js/src/kimchi.api.js +++ b/ui/js/src/kimchi.api.js @@ -1114,6 +1114,20 @@ var kimchi = { }); }, + getHostPCIDevices : function(suc, err) { + kimchi.requestJSON({ + url : kimchi.url + 'host/devices?_passthrough=true&_cap=pci', + type : 'GET', + contentType : 'application/json', + dataType : 'json', + resend : true, + success : suc, + error : err ? err : function(data) { + kimchi.message.error(data.responseJSON.reason); + } + }); + }, + getISCSITargets : function(server, port, suc, err) { server = encodeURIComponent(server); port = port ? '&_server_port='+encodeURIComponent(port) : ''; @@ -1144,6 +1158,47 @@ var kimchi = { }); }, + getVMPCIDevices : function(id, suc, err) { + kimchi.requestJSON({ + url : kimchi.url + 'vms/'+encodeURIComponent(id)+'/hostdevs', + type : 'GET', + contentType : 'application/json', + dataType : 'json', + resend : true, + success : suc, + error : err ? err : function(data) { + kimchi.message.error(data.responseJSON.reason); + } + }); + }, + + addVMPCIDevice : function(vm, device, suc, err) { + kimchi.requestJSON({ + url : kimchi.url + 'vms/'+ encodeURIComponent(vm) +'/hostdevs', + type : 'POST', + contentType : 'application/json', + dataType : 'json', + data : JSON.stringify(device), + success : suc, + error : err ? err : function(data) { + kimchi.message.error(data.responseJSON.reason); + } + }); + }, + + removeVMPCIDevice : function(vm, device, suc, err) { + kimchi.requestJSON({ + url : kimchi.url + 'vms/'+ encodeURIComponent(vm) +'/hostdevs/' + encodeURIComponent(device), + type : 'DELETE', + contentType : 'application/json', + dataType : 'json', + success : suc, + error : err ? err : function(data) { + kimchi.message.error(data.responseJSON.reason); + } + }); + }, + /** * Add a volume to a given storage pool. */ diff --git a/ui/js/src/kimchi.guest_edit_main.js b/ui/js/src/kimchi.guest_edit_main.js index c281289..030e112 100644 --- a/ui/js/src/kimchi.guest_edit_main.js +++ b/ui/js/src/kimchi.guest_edit_main.js @@ -359,6 +359,86 @@ kimchi.guest_edit_main = function() { }); }; + var setupPCIDevice = function(){ + kimchi.getHostPCIDevices(function(hostPCIs){ + kimchi.getVMPCIDevices(kimchi.selectedGuest, function(vmPCIs){ + kimchi.getCapabilities(function(result) { + var pciEnabled = result.kernel_vfio; + for(var i=0; i<hostPCIs.length; i++){ + var itemNode = $.parseHTML(kimchi.substitute($('#pci-tmpl').html(),{ + name: hostPCIs[i].name, + product: hostPCIs[i].product.description, + vendor: hostPCIs[i].vendor.description + })); + $(".body", "#form-guest-edit-pci").append(itemNode); + var iconClass = "ui-icon-plus"; + for(var j=0; j<vmPCIs.length; j++){ + if(hostPCIs[i].name==vmPCIs[j].name){ + iconClass = "ui-icon-minus"; + break; + } + } + pciEnabled || $("button", itemNode).remove(); + $("button", itemNode).button({ + icons: { primary: iconClass }, + text: false + }).click(function(){ + var obj = $(this); + if(obj.button("option", "icons").primary == "ui-icon-minus"){ + kimchi.removeVMPCIDevice(kimchi.selectedGuest, obj.parent().prop("id"), function(){ + kimchi.getVMPCIDevices(kimchi.selectedGuest, function(vmPCIs1){ + for(var k=0; k<hostPCIs.length; k++) { + $("button", "#" + hostPCIs[k].name).button("option", "icons", {primary: "ui-icon-plus"}); + } + for(var k=0; k<vmPCIs1.length; k++) { + $("button", "#" + vmPCIs1[k].name).button("option", "icons", {primary: "ui-icon-minus"}); + } + }); + filterNodes($("select", "#form-guest-edit-pci").val(), $("input", "#form-guest-edit-pci").val()); + }); + }else{ + kimchi.addVMPCIDevice(kimchi.selectedGuest, { name: obj.parent().prop("id") }, function(){ + kimchi.getVMPCIDevices(kimchi.selectedGuest, function(vmPCIs1){ + for(var k=0; k<vmPCIs1.length; k++) { + $("button", "#" + vmPCIs1[k].name).button("option", "icons", {primary: "ui-icon-minus"}); + } + }); + filterNodes($("select", "#form-guest-edit-pci").val(), $("input", "#form-guest-edit-pci").val()); + }); + } + }); + } + }); + }); + }); + var filterNodes = function(group, text){ + text = text.toLowerCase(); + $(".body", "#form-guest-edit-pci").children().each(function(){ + var textFilter = $(".name", this).text().toLowerCase().indexOf(text)!=-1; + textFilter = textFilter || $(".product", this).text().toLowerCase().indexOf(text)!=-1; + textFilter = textFilter || $(".vendor", this).text().toLowerCase().indexOf(text)!=-1; + var display = "none"; + var itemGroup = $("button", this).button("option", "icons").primary; + if(textFilter){ + if(group == "all"){ + display = ""; + }else if(group=="toAdd" && itemGroup=="ui-icon-plus"){ + display = "" + }else if(group == "added" && itemGroup=="ui-icon-minus"){ + display = "" + } + } + $(this).css("display", display); + }); + }; + $("select", "#form-guest-edit-pci").change(function(){ + filterNodes($(this).val(), $("input", "#form-guest-edit-pci").val()); + }); + $("input", "#form-guest-edit-pci").on("keyup", function() { + filterNodes($("select", "#form-guest-edit-pci").val(), $(this).val()); + }); + }; + var initContent = function(guest) { guest['icon'] = guest['icon'] || 'images/icon-vm.png'; $('#form-guest-edit-general').fillWithObject(guest); @@ -395,6 +475,7 @@ kimchi.guest_edit_main = function() { initStorageListeners(); setupInterface(); setupPermission(); + setupPCIDevice(); kimchi.topic('kimchi/vmCDROMAttached').subscribe(onAttached); kimchi.topic('kimchi/vmCDROMReplaced').subscribe(onReplaced); diff --git a/ui/pages/guest-edit.html.tmpl b/ui/pages/guest-edit.html.tmpl index 917b2e8..69b11a7 100644 --- a/ui/pages/guest-edit.html.tmpl +++ b/ui/pages/guest-edit.html.tmpl @@ -41,6 +41,9 @@ <li> <a href="#form-guest-edit-permission">$_("Permission")</a> </li> + <li> + <a href="#form-guest-edit-pci">$_("Host PCI Device")</a> + </li> </ul> <form id="form-guest-edit-general"> <fieldset class="guest-edit-fieldset"> @@ -138,6 +141,23 @@ </div> </div> </form> + <form id="form-guest-edit-pci" class="guest-edit-pci"> + <div class="filter"> + <span class="group"> + <select class="control"> + <option value="all">$_("All")</option> + <option value="toAdd">$_("To Add")</option> + <option value="added">$_("Added")</option> + </select><input type="text" class="control" placeholder="$_("filter")"> + </span> + </div> + <div class="header"> + <span class="cell name">$_("Name")</span> + <span class="cell product">$_("Product")</span> + <span class="cell vendor">$_("Vendor")</span> + </div> + <div class="body"></div> + </form> </div> </div> <footer> @@ -221,6 +241,14 @@ <label>{val}</label> </div> </script> +<script id="pci-tmpl" type="text/html"> +<div class="item" id="{name}"> + <span class="cell name" title="{name}">{name}</span> + <span class="cell product" title="{product}">{product}</span> + <span class="cell vendor" title="{vendor}">{vendor}</span> + <button></button> +</div> +</script> <script type="text/javascript"> kimchi.guest_edit_main(); </script>