[Users] Experience with low cost NFS-Storage as VM-Storage?

[Markus Stockhausen]

> Hi,
> 
>I'd like to ask around if someone does run oVirt
> with NFS backed Storage provided by simple servers (no SAN or NAS)
> and what your experience is so far?

We are running OVirt (still in test phase) on three self build equal Ubuntu
NFS servers over Infiniband (50 Euros for a ConnectX card + 50 Euros a cable +
400 Euros a switch). Due to massive bandwidth the bottleneck are the SATA 
disks and the RAID controller. I fixed some info at http://www.ovirt.org/Infiniband
Our soon to be replaced ESX infrastucture uses the same platform.

> In particular I'm interested what happens if there is a connection
> loss to the NFS-Volume.

You can be sure that the applications in your VMs will get stuck. They usually flush 
their write caches in regular intervals. This is directly translated into a NFS write
on the hypervisor. At least it should be configured that way otherwise you could
loose data in case of a crash. E.g. VM thinks data is persisent, but it was only cached
in the hypervisor RAM.

> How does this affect running vms and the compute nodes they run on?

The hypervisor has no problems at all (except you run a df command) and the 
VMs usually will continue their operation after NFS recovers. Nevertheless 
the are lot of timeout settings in the architecute stack that may stop a running
application inside the VM. Expect do have a lot of manual cleanup after a
NFS failure.

> I suspect they would first write their changes to RAM instead of virtual HDD.
> But once the RAM is full, does just the vm become unresponsive or
> does the whole compute node die?

That would be bad practice. See above.

> I couldn't test this yet myself, but my limited experience with
> NFS-Servers tells me, that they become unresponsive if they are under
> heavy load and I'd like to know how this affects the vms and
> computenode.

Defining heavy load is basically some kind of I/O calculation. A simple
example from our setup.

- Each NFS server has a RAID6 consisting of 14 SATA disks (7.2K)
- The top-loaded NFS server runs a minimum of 140 8K I/Os per second
- Assuming a random pattern we speak of 140 Read-Modify-Write cycles
- That translates to roughly to 6*140=840 I/Os per second.
- The 14 SATA disks offer a maximum of 14*90=1260 I/Os per second
- No wonder that the NFS server gives usually a 20% wait I/O usage.

> Thanks!

> PS: Bonus question: Does someone utilize the NFS-Servers also as
> computenodes ?

We spearated that stricly. Otherwise we would go with a Gluster FS.

Markus

P.S. We leave VMWare because live storage migration of OVirt VMs 
gives us the possibility to reduce the complexity on the storage side.
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: InterScan_Disclaimer.txt
URL: <http://lists.ovirt.org/pipermail/users/attachments/20140108/37284c5e/attachment-0001.txt>