The caveat with local storage is that I can only use the remaining free
space in /var/ for disk images. The result is the 1TB SSD has around
700GB remaining free space.
So I was wondering about simply passing through the nvme ssd (PCI) to the
guest, so the guest can utilise the fill SSD.
Are there any "gotcha's" with doing this other than the usual gpu
passthrough ones?
The caveat is that you cannot pass through a partial SSD, only a whole device. And as a
matter of fact, I'd say you can only pass through entire PCI(e) devices, so
traditional SCSI/SATA disks might not work individually, you'd have to pass through
the entire controller.
Effectively you'd "unplug" the NVMe from the host and plug it into the guest
and if that wasn't stopped by some sanity check, neither system would continue to run
for long, if that is your boot device (or the disk is used otherwise).
I know 3x more IOPS sound attractive, but is your workload really that disk bound? Perhaps
you'll need to think about some distributed memory cache.