6:49 a.m.
Hi,
We are looking at getting some GPU's for our servers and to use vGPU passthrough so
that our students can do some video renders on the VM's. Does anyone have good
experience with the Nvidia Quadro RTX6000 or RTX8000 and ovirt 4.3?
Thanks.
Kim
12:06 p.m.
The main issue isn't oVirt, but Nvidia's drivers inside a virtual machine: You use
'unapproved' GPUs in what Nvidia drivers recognise as a VM, they'll refuse to
load.
RTX series cards should be ok, I've tried K40 and P100 and they are just as fine with
oVirt pass-through.
V100 tests are still outstanding, but I am glad Matthias is reporting success.
I've been able to trick consumer GPUs into working with KVM on machines that I also
use for oVirt, but oVirt itself is constructing the XML files for KVM on-the-fly, so
you'd have to fiddle with the code that builds them: I haven't even found that
yet.
In terms of video rendering I've found the VirtualGL project (https://virtualgl.org/)
rather fascinating, which allowed me to run the unengine singularity benchmark and
"game" on a V100 GPU projected to a notebook hundreds of kilometers away...
The RTX series is supposed to enable partitioning of the GPUs, so several students can get
a slice each: Never tried, don't know if that requries extra drivers etc. The target
here is evidently remote CAD for aviation and defense, where money belts are quite a bit
wider than in education.
12:48 p.m.
IIRC oVirt 4.3 should have the basic hooks in place for mdev
passthrough. For nvidia this mean you need the vgpu drivers and a
license server. These licenses have a recurring cost.
AMD's solution uses SR-IOV, and requires a custom kernel module that is
not well tested YMMV.
You can also passthrough entire cards in ovirt without any drivers,
granted since the amount of GPU's you can stuff into a server is limited
this is probably not ideal. Nvidia blocks this on consumer level cards.
Intel also has a mdev solution that is built into mesa / the kernel
already, we have used it with VCA cards in the past, but perhaps the new
Intel GPU's support it as well? Intel is the only option that will allow
you to feed the framebuffer data back into spice so you can use the
ovirt console. All other options require you to create a remote session
to the guest directly.
On 2021-01-25 07:49, kim.kargaard(a)noroff.no wrote:
> Hi,
>
> We are looking at getting some GPU's for our servers and to use vGPU
> passthrough so that our students can do some video renders on the
> VM's. Does anyone have good experience with the Nvidia Quadro RTX6000
> or RTX8000 and ovirt 4.3?
>
> Thanks.
>
> Kim
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/XF4LZOEAC2Y...
3:24 a.m.
Thanks all for the feedback.
When we bought the servers, they put some Nvidia Quadro P4000's in, but I can't
seem to find anything that says they support vGPU.
I wasn't aware of the license server requirement for Nvidia, but the pricing isn't
too bad, so I can probably get that pushed through. Anyone aware of which AMD cards are
supported by oVirt? I have only found a list of Nvidia cards.
Thanks,
Kim
5:50 a.m.
On Tue, Jan 26, 2021 at 10:26 AM <kim.kargaard(a)noroff.no> wrote:
Hello,
updated list of supported GPUs for vGPU functionality should be this one:
https://docs.nvidia.com/grid/gpus-supported-by-vgpu.html
and P4000 seems not to be present
So you could use the other supported method, with passthrough Gpu,
following what described here for qemu command line:
https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#using...
and driving oVirt to do so with vdsm hook qemucmdline
[root@ov200 ~]# dnf info vdsm-hook-qemucmdline.noarch
Last metadata expiration check: 2:48:59 ago on Tue 26 Jan 2021 09:58:06 AM
CET.
Available Packages
Name : vdsm-hook-qemucmdline
Version : 4.40.40
Release : 1.el8
Architecture : noarch
Size : 15 k
Source : vdsm-4.40.40-1.el8.src.rpm
Repository : ovirt-4.4
Summary : QEMU cmdline hook for VDSM
URL : http://www.ovirt.org/develop/developer-guide/vdsm/vdsm/
License : GPLv2+
Description : Provides support for injecting QEMU cmdline via VDSM hook.
: It exploits libvirt's qemu:commandline facility available in
the
: qemu xml namespace.
[root@ov200 ~]#
I donna if the doc here is up to date, not tested lately:
https://www.ovirt.org/develop/developer-guide/vdsm/hook/qemucmdline.html
I think the persist command in case of node is not necessary with NG any
more
Developers could confirm and eventually point to the correct page for the
hook guide?
Gianluca
10:50 a.m.
On Tue, Jan 26, 2021 at 5:30 PM Michal Skrivanek <
michal.skrivanek(a)redhat.com> wrote:
My suggestion for the hook was only for GPU passthrough, not being P4000
supported for vGPU.
Good to know that GPU passthrough is supported/integrated. Do you have a
description of the workflow for doing it without a hook? Edit VM? Add
device?
Thanks,
Gianluca
2:23 a.m.
On Wed, Jan 27, 2021 at 9:14 AM <kim.kargaard(a)noroff.no> wrote:
I would also be interested to know that :)
Ok, I think I found at least for Nvidia. You can follow what described for
RHV:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/...
In the same manual there are also instructions for vGPU.
There is also the guide for 4.3:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/...
Gianluca
12:20 p.m.
Again, that's the easy part that works just fine.
It's afterwards when you try to install the Nvidia driver when there might be trouble
because the driver refused to load on GPUs that aren't permitted to operate inside a
VM, quite independent of the hypervisor (same issue with VMware, VirtualBox, KVM).
For that there are tricks you'll find on the Internet on how to cheat the Nvidia
driver into believing it's running on a physical machine but on KVM that implies
editing the XML config... which in the case of oVirt seems to be generated at startup.
So I guess the hooking mechanism can be used to take care of that, but I've not
progressed to that stage myself, mostly because the GPUs I use in the corporate lab are
whitelisted by Nvidia.
And just to give a little credit: Pass-through works very well with oVirt 4.3, only moving
(inactive) VMs from host-to-host is a little more involved, while GPU live-migration
unfortunately is out of the question. That makes perfect sense for NICs and storage
adapters, for GPUs one might argue that their state could be migrated, too. But Linux
doesn't really understand heterogenous accelerators yet, so oVirt can't do better.
9:27 a.m.
yes. you can use that.
shouldn’t be too hard, you can use any of the existing hooks (even old ones) as a starting
point. I think (I haven’t looked at that for a long time) that smbios modification may be
enough, so there’s [1] right away. If not, it should be straightforward to adapt
it’s a generic PCI VFIO passthrough, there’s nothing specific about GPU.
Migrateable GPUs are not generic and will require some work, mostly in kernel/drivers.
nvidia is working on it, but it’s hard to say when that’s going to be ready, and as always
i doubt they’ll make it available for older/lowend cards. But then again maybe with some
tricks…:)
Thanks,
michal
[1] https://github.com/oVirt/vdsm/tree/master/vdsm_hooks/smbios
1526
days inactive
1529
days old
12 comments
7 participants
participants (7)
-
Alex McWhirter -
Gianluca Cecchi -
kim.kargaard＠noroff.no -
Matthias Leopold -
Michal Skrivanek -
Shantur Rathore -
Thomas Hoberg