That's exactly the direction I originally understood oVirt would go, with the ability
to run VMs and container side-by-side on the bare metal or nested with containers inside
VMs for stronger resource or security isolation and network virtualization. To me it
sounded especially attractive with an HCI underpinning so you could deploy it also in the
field with small 3 node clusters.
But combining all those features evidently comes at too high a cost for all the
integration and the customer base is either too small or too poor: the cloud players are
all out on making sure you no longer run any hardware and then it's really just about
pushing your applications there as cloud native or "IaaS" compatible as needed.
E.g. I don't see PCI pass-through coming to kubevirt to enable GPU use, because it
ties the machine to a specific host and goes against the grain of K8 as I understand it.
Memory overcommit is quite funny, really, because it's the same issue as the original
virtual memory: essentially you lie to your consumer about the resources available and
then swap pages forth and back in an attempt to make all your consumers happy. It was
processes for virtual memory, it's VMs now for the hypervisor and in both cases
it's about the consumer and the provider not continously negotiating for the resources
they need and the price they are willing to pay.
That negotiation is always better at the highest level of abstraction, the application
itself, which why implementing it at the lower levels (e.g. VMs) becomes less useful and
needed.
And then there is technology like CXL which essentially turns RAM in to a fabric and your
local CPU will just get RAM from another piece of hardware when your application needs
more RAM and is willing to pay the premium something will charge for it.
With that type of hardware much of what hypervisors used to do goes into DPUs/IPUs and
CPUs are just running applications making hypercalls. The kernel is just there to
bootstrap.
Not sure we'll see that type of hardware at home or in the edge, though...