On Mon, Feb 21, 2022 at 12:27 PM Thomas Hoberg <thomas(a)hoberg.net&gt; wrote: technically it's already there: https://kubevirt.io/user-guide/virtual_machines/host-devices/

Well, I guess it's there, mostly because they didn't have to do anything new, it's part of KVM/libvirt and more inherited than added. The main reason I "don't see it coming" is that may create more problems than it solves. To my understanding K8 is all about truly elastic workloads, including "mobility" to avoid constraints (including memory overcommit). Mobility in quotes, because I don't even know if it migrates containers or just shuts instances down in one place and launches them in another: migration itself has a significant cost after all. But if it were to migrate them (e.g. via CRIU for containers and "vMotion" for VMs) it would then to also have to understand (via KubeVirt), which devices are tied, because they use a device that has too big a state (e.g. a multi-gig CUDA workloads), a hard physical dependence (e.g. USB with connected devices) or something that could move with the VM (e.g. SR-IOV FC/NIC/INF with a fabric that can be re-configured to match or is also virtualized). A proper negotiation between the not-so-dynamic physically available assets of the DC and the much more dynamic resources required by the application are the full scope of a virt-stack/k8 hybrid, encompassing a DC/Cloud-OS (infrastructure) and K8 (platform) aspects. While I'd love to have that, I can see how that won't be maintained by anyone as a full free-to-use open-souce turn-key solution.

One of these days I'll have to dig deep and see what it does. "Scheduling" can encompass quite a few activities and I don't know which of them K8 covers. Batch scheduling (also Singularity/HPC) type scheduling involves also the creation (and thus consumption) of RAM/storage/GPUs, so real instances/reservations are created, which in the case of pass-through would include some hard dependencies. Normal OS scheduling is mostly CPU, while processes and stroage are alrady there and occupy resources and could find an equivalent in traffic steering, where the number of nodes that receive traffic is expanded or reduced. K8 to my understanding would do the traffic steering as a minimum and then have actions for instance creation and deletions. But given a host with hybrid loads, some with tied resources, others generic without: to manage the decisions/allocations properly you need to come up with a plan that includes 1. Given a capacity bottleneck on a host, do I ask the lower-layer (DC-OS) to create additional containers elsewhere and shut down the local ones or do I migrate the running one on a new host? 2. Given a capacity underutilization on a host, how to go best about shutting down hosts, that aren't going to be needed for the next hours in a way where the migration cost do not exceed the power savings? To my naive current understanding virt-stacks won't create and shut-down VMs, their typical (or only?) load management instrument is VM migration. Kubernetes (and Docker swarm etc.) won't migrate node instances (nor VMs), but create and destory them to manage load. At large scales (scale out) this swarm approach is obviously better, migration creates too much of an overhead. In the home domain of the virt-stacks (scale in), live migration is perhaps necessary, because the application stacks aren't ready to deal with instance destruction without service disruption or just because it is rare enough to be cheaper than instance re-creation. In the past it was more clear cut, because there was no live migration support for containers. But with CRIU (and its predecessors in OpenVZ), that could be done just as seamless as with VMs. And instance creation/deletions were more like fail-over scenarios, where (rare) service disruptions were accepted. Today the two approaches can more easily be mingled but they don't easily mix yet, and the negotiating element between them is missing. Just understanding the two approaches and how they mix is already filling the brain capacity I can give them. Operating that mix is currently quite beyond the fact that it's only a small part of my job. Gleichfalls!

On Mon, Feb 21, 2022 at 12:27 PM Thomas Hoberg <thomas(a)hoberg.net&gt; wrote: I think in general a big part of the industry is going down the path of moving most things behind the k8s API/resource model. This means different things for different companies. For instance vmware keeps its traditional virt-stack, adding additional k8s apis in front of it, and crossing the bridges to k8s clusters behind the scenes to get a unified view, while other parts are choosing k8s (be it vanilla k8s, openshift, harvester, ...) and then take for instance KubeVirt to deploy additional k8s clusters on top of it, unifying the stack this way. It is definitely true that k8s works significantly different to other solutions like oVirt or OpenStack, but once you get into it, I think one would be surprised how simple the architecture of k8s actually is, and also how little resources core k8s actually takes. Having said that, as an ex oVirt engineer I would be glad to see oVirt continue to thrive. The simplicity of oVirt was always appealing to me. Best regards, Roman