[ovirt-devel] Re: device compatibility interface for live migration with assigned devices

On Thu, Jul 16, 2020 at 12:16:26PM +0800, Jason Wang wrote: not familiar with the devlink. will do some research of it. but the VFIO live migration in itself is essentially a big opaque stream to userspace. yes, in another reply, Alex proposed to use an interface in json format. I guess we can define something like { "self" : [ { "pciid" : "8086591d", "driver" : "i915", "gvt-version" : "v1", "mdev_type" : "i915-GVTg_V5_2", "aggregator" : "1", "pv-mode" : "none", } ], "compatible" : [ { "pciid" : "8086591d", "driver" : "i915", "gvt-version" : "v1", "mdev_type" : "i915-GVTg_V5_2", "aggregator" : "1" "pv-mode" : "none", }, { "pciid" : "8086591d", "driver" : "i915", "gvt-version" : "v1", "mdev_type" : "i915-GVTg_V5_4", "aggregator" : "2" "pv-mode" : "none", }, { "pciid" : "8086591d", "driver" : "i915", "gvt-version" : "v2", "mdev_type" : "i915-GVTg_V5_4", "aggregator" : "2" "pv-mode" : "none, ppgtt, context", } ... ] } But as those fields are mostly vendor specific, the userspace can only do simple string comparing, I guess the list would be very long as it needs to enumerate all possible targets. also, in some fileds like "gvt-version", is there a simple way to express things like v2+? If the userspace can read this interface both in src and target and check whether both src and target are in corresponding compatible list, I think it will work for us. But still, kernel should not rely on userspace's choice, the opaque compatibility string is still required in kernel. No matter whether it would be exposed to userspace as an compatibility checking interface, vendor driver would keep this part of code and embed the string into the migration stream. so exposing it as an interface to be used by libvirt to do a safety check before a real live migration is only about enabling the kernel part of check to happen ahead. Thanks Yan > > > > for a Intel vGPU, string format can be defined like > > "parent device PCI ID" + "version of gvt driver" + "mdev type" + "aggregator count". > > > > for an NVMe VF connecting to a remote storage. it could be > > "PCI ID" + "driver version" + "configured remote storage URL" > > > > for a QAT VF, it may be > > "PCI ID" + "driver version" + "supported encryption set". > > > > (to avoid namespace confliction from each vendor, we may prefix a driver name to > > each migration_version string. e.g. i915-v1-8086-591d-i915-GVTg_V5_8-1) > > > > > > (2) backgrounds > > > > The reason we hope the migration_version string is opaque to the userspace > > is that it is hard to generalize standard comparing fields and comparing > > methods for different devices from different vendors. > > Though userspace now could still do a simple string compare to check if > > two devices are compatible, and result should also be right, it's still > > too limited as it excludes the possible candidate whose migration_version > > string fails to be equal. > > e.g. an MDEV with mdev_type_1, aggregator count 3 is probably compatible > > with another MDEV with mdev_type_3, aggregator count 1, even their > > migration_version strings are not equal. > > (assumed mdev_type_3 is of 3 times equal resources of mdev_type_1). > > > > besides that, driver version + configured resources are all elements demanding > > to take into account. > > > > So, we hope leaving the freedom to vendor driver and let it make the final decision > > in a simple reading from source side and writing for test in the target side way. > > > > > > we then think the device compatibility issues for live migration with assigned > > devices can be divided into two steps: > > a. management tools filter out possible migration target devices. > > Tags could be created according to info from product specification. > > we think openstack/ovirt may have vendor proprietary components to create > > those customized tags for each product from each vendor. > > e.g. > > for Intel vGPU, with a vGPU(a MDEV device) in source side, the tags to > > search target vGPU are like: > > a tag for compatible parent PCI IDs, > > a tag for a range of gvt driver versions, > > a tag for a range of mdev type + aggregator count > > > > for NVMe VF, the tags to search target VF may be like: > > a tag for compatible PCI IDs, > > a tag for a range of driver versions, > > a tag for URL of configured remote storage. > > > > b. with the output from step a, openstack/ovirt/libvirt could use our proposed > > device migration compatibility interface to make sure the two devices are > > indeed live migration compatible before launching the real live migration > > process to start stream copying, src device stopping and target device > > resuming. > > It is supposed that this step would not bring any performance penalty as > > -in kernel it's just a simple string decoding and comparing > > -in openstack/ovirt, it could be done by extending current function > > check_can_live_migrate_destination, along side claiming target resources.[1] > > > > > > [1] https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/lib... > > > > Thanks > > Yan > > >