I've been able to trick consumer GPUs into working with KVM on machines that I also use for oVirt, but oVirt itself is constructing the XML files for KVM on-the-fly, so you'd have to fiddle with the code that builds them: I haven't even found that yet.

You could use qemu_cmdline VDSM hook to provide custom qemu command line params for your VM or you could write your own hook to change libvirt xml before the VM is powered on. On Mon, Jan 25, 2021 at 7:03 PM Alex McWhirter <alex@triadic.us> wrote:

IIRC oVirt 4.3 should have the basic hooks in place for mdev passthrough. For nvidia this mean you need the vgpu drivers and a license server. These licenses have a recurring cost.

AMD's solution uses SR-IOV, and requires a custom kernel module that is not well tested YMMV.

You can also passthrough entire cards in ovirt without any drivers, granted since the amount of GPU's you can stuff into a server is limited this is probably not ideal. Nvidia blocks this on consumer level cards.

Intel also has a mdev solution that is built into mesa / the kernel already, we have used it with VCA cards in the past, but perhaps the new Intel GPU's support it as well? Intel is the only option that will allow you to feed the framebuffer data back into spice so you can use the ovirt console. All other options require you to create a remote session to the guest directly.

On 2021-01-25 07:49, kim.kargaard@noroff.no wrote:

...

Hi,

We are looking at getting some GPU's for our servers and to use vGPU passthrough so that our students can do some video renders on the VM's. Does anyone have good experience with the Nvidia Quadro RTX6000 or RTX8000 and ovirt 4.3?

Thanks.

Kim _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:

https://lists.ovirt.org/archives/list/users@ovirt.org/message/XF4LZOEAC2YTRH... _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PY4ITOIWACUWHY...

On Tue, Jan 26, 2021 at 10:26 AM <kim.kargaard@noroff.no> wrote:

Thanks all for the feedback.

When we bought the servers, they put some Nvidia Quadro P4000's in, but I can't seem to find anything that says they support vGPU.

I wasn't aware of the license server requirement for Nvidia, but the pricing isn't too bad, so I can probably get that pushed through. Anyone aware of which AMD cards are supported by oVirt? I have only found a list of Nvidia cards.

Thanks,

Kim

Hello, updated list of supported GPUs for vGPU functionality should be this one: https://docs.nvidia.com/grid/gpus-supported-by-vgpu.html and P4000 seems not to be present So you could use the other supported method, with passthrough Gpu, following what described here for qemu command line: https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#using-gp... and driving oVirt to do so with vdsm hook qemucmdline [root@ov200 ~]# dnf info vdsm-hook-qemucmdline.noarch Last metadata expiration check: 2:48:59 ago on Tue 26 Jan 2021 09:58:06 AM CET. Available Packages Name : vdsm-hook-qemucmdline Version : 4.40.40 Release : 1.el8 Architecture : noarch Size : 15 k Source : vdsm-4.40.40-1.el8.src.rpm Repository : ovirt-4.4 Summary : QEMU cmdline hook for VDSM URL : http://www.ovirt.org/develop/developer-guide/vdsm/vdsm/ License : GPLv2+ Description : Provides support for injecting QEMU cmdline via VDSM hook. : It exploits libvirt's qemu:commandline facility available in the : qemu xml namespace. [root@ov200 ~]# I donna if the doc here is up to date, not tested lately: https://www.ovirt.org/develop/developer-guide/vdsm/hook/qemucmdline.html I think the persist command in case of node is not necessary with NG any more Developers could confirm and eventually point to the correct page for the hook guide? Gianluca

On Tue, Jan 26, 2021 at 5:30 PM Michal Skrivanek < michal.skrivanek@redhat.com> wrote:

why a hook? we have a “native” ovirt support for both GPU and vGPU since cca 4.1. There was a hook initially, but that’s no longer needed…since 4.2 iirc I think we also tested some Intel’s GTV-g card...

My suggestion for the hook was only for GPU passthrough, not being P4000 supported for vGPU. Good to know that GPU passthrough is supported/integrated. Do you have a description of the workflow for doing it without a hook? Edit VM? Add device? Thanks, Gianluca

On Wed, Jan 27, 2021 at 9:14 AM <kim.kargaard@noroff.no> wrote:

I would also be interested to know that :)

Ok, I think I found at least for Nvidia. You can follow what described for RHV: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/htm... In the same manual there are also instructions for vGPU. There is also the guide for 4.3: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/htm... Gianluca

On 27. 1. 2021, at 19:20, Thomas Hoberg <thomas@hoberg.net> wrote:

...

On Wed, Jan 27, 2021 at 9:14 AM <kim.kargaard(a)noroff.no> wrote:

Ok, I think I found at least for Nvidia. You can follow what described for RHV: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/...

In the same manual there are also instructions for vGPU.

There is also the guide for 4.3: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/...

Gianluca

Again, that's the easy part that works just fine. It's afterwards when you try to install the Nvidia driver when there might be trouble because the driver refused to load on GPUs that aren't permitted to operate inside a VM, quite independent of the hypervisor (same issue with VMware, VirtualBox, KVM).

For that there are tricks you'll find on the Internet on how to cheat the Nvidia driver into believing it's running on a physical machine but on KVM that implies editing the XML config... which in the case of oVirt seems to be generated at startup.

yes. you can use that.

So I guess the hooking mechanism can be used to take care of that, but I've not progressed to that stage myself, mostly because the GPUs I use in the corporate lab are whitelisted by Nvidia.

shouldn’t be too hard, you can use any of the existing hooks (even old ones) as a starting point. I think (I haven’t looked at that for a long time) that smbios modification may be enough, so there’s [1] right away. If not, it should be straightforward to adapt

And just to give a little credit: Pass-through works very well with oVirt 4.3, only moving (inactive) VMs from host-to-host is a little more involved, while GPU live-migration unfortunately is out of the question. That makes perfect sense for NICs and storage adapters, for GPUs one might argue that their state could be migrated, too. But Linux doesn't really understand heterogenous accelerators yet, so oVirt can't do better.

it’s a generic PCI VFIO passthrough, there’s nothing specific about GPU. Migrateable GPUs are not generic and will require some work, mostly in kernel/drivers. nvidia is working on it, but it’s hard to say when that’s going to be ready, and as always i doubt they’ll make it available for older/lowend cards. But then again maybe with some tricks…:) Thanks, michal [1] https://github.com/oVirt/vdsm/tree/master/vdsm_hooks/smbios

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/S3DBQHTQDOZ6AS...