[Kimchi-devel] [PATCH v12 2/6] Host device passthrough: List eligible device to passthrough

Aline Manera alinefm at linux.vnet.ibm.com
Mon Oct 13 15:42:36 UTC 2014


Reviewed-by: Aline Manera <alinefm at linux.vnet.ibm.com>

On 10/08/2014 06:08 AM, Zhou Zheng Sheng wrote:
> This patch adds a '_passthrough=true' filter to /host/devices, so it can
> filter and shows all devices eligible to passthrough to guest.
> Theoretically, all PCI, USB and SCSI devices can be assigned to guest
> directly.
>
> Linux kernel is able to recognize the host IOMMU group layout. If two
> PCI devices are in the same IOMMU group, it means there are possible
> interconnections between the devices, and the devices can talk to each
> other bypassing IOMMU. This implies isolation is not pefect between those
> devices, so all devices in a IOMMU group must be assigned to guest
> together. On host that recognizes IOMMU groups, by accessing the URI
> /host/devices?_passthrough_affected_by=DEVICE_NAME, it returns a list
> containing the devices in the same IOMMU group as DEVICE_NAME, and all
> of the children devices of them. So the front-end can show all the
> affected devices to user, and it helps the user to determine which
> host devices are to be assigned to guest.
>
> How to test:
>
> List all types of devices to passthrough
>    curl -k -u root -H "Content-Type: application/json" \
>      -H "Accept: application/json" \
>      'https://127.0.0.1:8001/host/devices?_passthrough=true'
>
> List all eligible PCI devices to passthrough
>    /host/devices?_passthrough=true&_cap=pci
>
> List all USB devices to passthrough
>    /host/devices?_passthrough=true&_cap=usb_device
>
> List all SCSI devices to passthrough
>    /host/devices?_passthrough=true&_cap=scsi
>
> List devices in the same IOMMU group as pci_0000_00_19_0
>    /host/devices?_passthrough_affected_by=pci_0000_00_19_0
>
> v1:
>    v1 series does not contain this patch.
>
> v2:
>    Deal with calculation "leaf" device and "affected" device.
>
> v5:
>    Change _passthrough=1 to _passthrough=true in the URI scheme. Filter
> PCI devices according the PCI class.
>
> v6:
>    Don't passthrough PCI device of class code 07. In modern
> x86 machine, it's possible that
> "6 Series/C200 Series Chipset Family MEI Controller" and
> "6 Series/C200 Series Chipset Family KT Controller"
> are of this class code. These two devices are not suitable to
> passthrough to guest. We don't have simple and reliable way to
> distinguish normal serial controller and host chipset XXX controller.
> This type of PCI devices also include various serial, parallel, modem,
> communication controller. Serial and parallel controllers can be
> re-direct from ttyS0 to QEMU's pty using socat, and there is little
> performance benefit to directly assign to guest. So it'k ok not to
> passththrough PCI device of class code 07.
>
> v8:
>    Use a new flag filter "_passthrough_group_by"
>      /host/devices?_passthrough_group_by=pci_XXX
>    instead of using sub-collection
>      /host/devices/pci_XXX/passthrough_affected_devices
>
> v9:
>    Use the same LibvirtConnection object as the Model, so as to avoid
>    connection exhausting.
>
> v10:
>    Adapt to RHEL 6. RHEL 6 does not provide iommu group information in
>    sysfs. For now we just ignore this error and live with it. The device
>    passthrough for PCI devices will not work, but the basic devices
>    informations are still provided to the user. In future we'll develope
>    code to gather iommu group information.
>
> v11:
>    In previous commits, we don't allow to passthrough device with children,
>    and only passthrough the children. It proves it's inflexible and less
>    useful. The PCI class code white list also filter out too much types of
>    devices, and there is no way to cleanly differenciate devices suitable
>    to passthrough using class code.
>
>    In this patch, we allow Kimchi to passthrough a parent PCI device. The
>    front-end can use the existing "_passthrough_group_by" filter to list
>    the affected children devices and other PCI devices in the same group,
>    as well as the children of those devices. We also drop the class code
>    white list, and filter out only PCI bridge and video cards.
>
>    libvirt uses domain, bus, slot and function to encode the device name,
>    so sorting the device name results the same effect as sorting based on
>    domain:bus:slot:function. This patch sorts all the devices based on the
>    name.
>
>    When an SCSI adapter is assigned to virtual machine, the previous
>    node device scsi_host and scsi_target become stale for the host machine.
>    Unfortunately, libvirt only removes the invalid scsi_host device, but
>    not scsi_target. When Kimchi is looking up the parent scsi_host of a
>    scsi_target, the scsi_host device actually does not exist, and it
>    explodes.
>
>    This patch catches such error and ignores those devices without a valid
>    parent device.
>
> v12:
>    Coding style improvements. Use "None" instead of "-1".
>
> Signed-off-by: Zhou Zheng Sheng <zhshzhou at linux.vnet.ibm.com>
> ---
>   src/kimchi/i18n.py          |   1 +
>   src/kimchi/model/host.py    |  33 ++++++++++--
>   src/kimchi/model/hostdev.py | 121 ++++++++++++++++++++++++++++++++++++++++++--
>   3 files changed, 147 insertions(+), 8 deletions(-)
>
> diff --git a/src/kimchi/i18n.py b/src/kimchi/i18n.py
> index 1b543ce..98adc46 100644
> --- a/src/kimchi/i18n.py
> +++ b/src/kimchi/i18n.py
> @@ -233,6 +233,7 @@ messages = {
>       "KCHHOST0001E": _("Unable to shutdown host machine as there are running virtual machines"),
>       "KCHHOST0002E": _("Unable to reboot host machine as there are running virtual machines"),
>       "KCHHOST0003E": _("Node device '%(name)s' not found"),
> +    "KCHHOST0004E": _("Conflicting flag filters specified."),
>
>       "KCHPKGUPD0001E": _("No packages marked for update"),
>       "KCHPKGUPD0002E": _("Package %(name)s is not marked to be updated."),
> diff --git a/src/kimchi/model/host.py b/src/kimchi/model/host.py
> index 7d7cd66..5d31809 100644
> --- a/src/kimchi/model/host.py
> +++ b/src/kimchi/model/host.py
> @@ -32,8 +32,9 @@ from kimchi import disks
>   from kimchi import netinfo
>   from kimchi import xmlutils
>   from kimchi.basemodel import Singleton
> -from kimchi.exception import InvalidOperation, NotFoundError, OperationFailed
>   from kimchi.model import hostdev
> +from kimchi.exception import InvalidOperation, InvalidParameter
> +from kimchi.exception import NotFoundError, OperationFailed
>   from kimchi.model.config import CapabilitiesModel
>   from kimchi.model.tasks import TaskModel
>   from kimchi.model.vms import DOM_STATE_MAP
> @@ -299,10 +300,28 @@ class DevicesModel(object):
>           except AttributeError:
>               self.cap_map['fc_host'] = None
>
> -    def get_list(self, _cap=None):
> +    def get_list(self, _cap=None, _passthrough=None,
> +                 _passthrough_affected_by=None):
> +        if _passthrough_affected_by is not None:
> +            # _passthrough_affected_by conflicts with _cap and _passthrough
> +            if (_cap, _passthrough) != (None, None):
> +                raise InvalidParameter("KCHHOST0004E")
> +            return sorted(
> +                self._get_passthrough_affected_devs(_passthrough_affected_by))
> +
>           if _cap == 'fc_host':
> -            return self._get_devices_fc_host()
> -        return self._get_devices_with_capability(_cap)
> +            dev_names = self._get_devices_fc_host()
> +        else:
> +            dev_names = self._get_devices_with_capability(_cap)
> +
> +        if _passthrough is not None and _passthrough.lower() == 'true':
> +            conn = self.conn.get()
> +            passthrough_names = [
> +                dev['name'] for dev in hostdev.get_passthrough_dev_infos(conn)]
> +            dev_names = list(set(dev_names) & set(passthrough_names))
> +
> +        dev_names.sort()
> +        return dev_names
>
>       def _get_devices_with_capability(self, cap):
>           conn = self.conn.get()
> @@ -314,6 +333,12 @@ class DevicesModel(object):
>                   return []
>           return [name.name() for name in conn.listAllDevices(cap_flag)]
>
> +    def _get_passthrough_affected_devs(self, dev_name):
> +        conn = self.conn.get()
> +        info = DeviceModel(conn=self.conn).lookup(dev_name)
> +        affected = hostdev.get_affected_passthrough_devices(conn, info)
> +        return [dev_info['name'] for dev_info in affected]
> +
>       def _get_devices_fc_host(self):
>           conn = self.conn.get()
>           # Libvirt < 1.0.5 does not support fc_host capability
> diff --git a/src/kimchi/model/hostdev.py b/src/kimchi/model/hostdev.py
> index 103c1e7..63cdb21 100644
> --- a/src/kimchi/model/hostdev.py
> +++ b/src/kimchi/model/hostdev.py
> @@ -17,7 +17,9 @@
>   # License along with this library; if not, write to the Free Software
>   # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301 USA
>
> +import os
>   from pprint import pformat
> +from pprint import pprint
>
>   from kimchi.model.libvirtconnection import LibvirtConnection
>   from kimchi.utils import kimchi_log
> @@ -36,7 +38,13 @@ def _get_dev_info_tree(dev_infos):
>           if dev_info['parent'] is None:
>               root = dev_info
>               continue
> -        parent = devs[dev_info['parent']]
> +
> +        try:
> +            parent = devs[dev_info['parent']]
> +        except KeyError:
> +            kimchi_log.error('Parent %s of device %s does not exist.',
> +                             dev_info['parent'], dev_info['name'])
> +            continue
>
>           try:
>               children = parent['children']
> @@ -47,6 +55,109 @@ def _get_dev_info_tree(dev_infos):
>       return root
>
>
> +def _is_pci_qualified(pci_dev):
> +    # PCI bridge is not suitable to passthrough
> +    # KVM does not support passthrough graphic card now
> +    blacklist_classes = (0x030000, 0x060000)
> +
> +    with open(os.path.join(pci_dev['path'], 'class')) as f:
> +        pci_class = int(f.readline().strip(), 16)
> +
> +    if pci_class & 0xff0000 in blacklist_classes:
> +        return False
> +
> +    return True
> +
> +
> +def get_passthrough_dev_infos(libvirt_conn):
> +    ''' Get devices eligible to be passed through to VM. '''
> +
> +    def is_eligible(dev):
> +        return dev['device_type'] in ('usb_device', 'scsi') or \
> +            (dev['device_type'] == 'pci' and _is_pci_qualified(dev))
> +
> +    dev_infos = _get_all_host_dev_infos(libvirt_conn)
> +
> +    return [dev_info for dev_info in dev_infos if is_eligible(dev_info)]
> +
> +
> +def _get_same_iommugroup_devices(dev_infos, device_info):
> +    dev_dict = dict([(dev_info['name'], dev_info) for dev_info in dev_infos])
> +
> +    def get_iommu_group(dev_info):
> +        # Find out the iommu group of a given device.
> +        # Child device belongs to the same iommu group as the parent device.
> +        try:
> +            return dev_info['iommuGroup']
> +        except KeyError:
> +            pass
> +
> +        parent = dev_info['parent']
> +        while parent is not None:
> +            try:
> +                parent_info = dev_dict[parent]
> +            except KeyError:
> +                kimchi_log.error("Parent %s of device %s does not exist",
> +                                 parent, dev_info['name'])
> +                break
> +
> +            try:
> +                iommuGroup = parent_info['iommuGroup']
> +            except KeyError:
> +                pass
> +            else:
> +                return iommuGroup
> +
> +            parent = parent_info['parent']
> +
> +        return None
> +
> +    iommu_group = get_iommu_group(device_info)
> +
> +    if iommu_group is None:
> +        return []
> +
> +    return [dev_info for dev_info in dev_infos
> +            if dev_info['name'] != device_info['name'] and
> +            get_iommu_group(dev_info) == iommu_group]
> +
> +
> +def _get_children_devices(dev_infos, device_info):
> +    def get_children_recursive(parent):
> +        try:
> +            children = parent['children']
> +        except KeyError:
> +            return []
> +
> +        result = []
> +        for child in children:
> +            result.append(child)
> +            result.extend(get_children_recursive(child))
> +
> +        return result
> +
> +    # Annotate every the dev_info element with children information
> +    _get_dev_info_tree(dev_infos)
> +
> +    for dev_info in dev_infos:
> +        if dev_info['name'] == device_info['name']:
> +            return get_children_recursive(dev_info)
> +
> +    return []
> +
> +
> +def get_affected_passthrough_devices(libvirt_conn, passthrough_dev):
> +    dev_infos = _get_all_host_dev_infos(libvirt_conn)
> +
> +    group_devices = _get_same_iommugroup_devices(dev_infos, passthrough_dev)
> +    if not group_devices:
> +        # On host without iommu group support, the affected devices should
> +        # at least include all children devices
> +        group_devices.extend(_get_children_devices(dev_infos, passthrough_dev))
> +
> +    return group_devices
> +
> +
>   def get_dev_info(node_dev):
>       ''' Parse the node device XML string into dict according to
>       http://libvirt.org/formatnode.html.
> @@ -168,8 +279,7 @@ def _get_usb_device_dev_info(info):
>
>
>   # For test and debug
> -def _print_host_dev_tree():
> -    libvirt_conn = LibvirtConnection('qemu:///system').get()
> +def _print_host_dev_tree(libvirt_conn):
>       dev_infos = _get_all_host_dev_infos(libvirt_conn)
>       root = _get_dev_info_tree(dev_infos)
>       if root is None:
> @@ -207,4 +317,7 @@ def _format_dev_node(node):
>
>
>   if __name__ == '__main__':
> -    _print_host_dev_tree()
> +    libvirt_conn = LibvirtConnection('qemu:///system').get()
> +    _print_host_dev_tree(libvirt_conn)
> +    print 'Eligible passthrough devices:'
> +    pprint(get_passthrough_dev_infos(libvirt_conn))




More information about the Kimchi-devel mailing list