This is a multi-part message in MIME format. ------=_NextPartTM-000-dc553fe8-e084-4205-a5f2-e4e48c40f662 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable =0A= We are running OVirt (still in test phase) on three self build equal Ubuntu= =0A= NFS servers over Infiniband (50 Euros for a ConnectX card + 50 Euros a cabl= e +=0A= 400 Euros a switch). Due to massive bandwidth the bottleneck are the SATA = =0A= disks and the RAID controller. I fixed some info at http://www.ovirt.org/In= finiband=0A= Our soon to be replaced ESX infrastucture uses the same platform.=0A= =0A= =0A= You can be sure that the applications in your VMs will get stuck. They usua= lly flush =0A= their write caches in regular intervals. This is directly translated into a= NFS write=0A= on the hypervisor. At least it should be configured that way otherwise you = could=0A= loose data in case of a crash. E.g. VM thinks data is persisent, but it was= only cached=0A= in the hypervisor RAM.=0A= =0A= =0A= The hypervisor has no problems at all (except you run a df command) and the= =0A= VMs usually will continue their operation after NFS recovers. Nevertheless = =0A= the are lot of timeout settings in the architecute stack that may stop a ru= nning=0A= application inside the VM. Expect do have a lot of manual cleanup after a= =0A= NFS failure.=0A= =0A= HDD.=0A= =0A= That would be bad practice. See above.=0A= =0A= =0A= Defining heavy load is basically some kind of I/O calculation. A simple=0A= example from our setup.=0A= =0A= - Each NFS server has a RAID6 consisting of 14 SATA disks (7.2K)=0A= - The top-loaded NFS server runs a minimum of 140 8K I/Os per second=0A= - Assuming a random pattern we speak of 140 Read-Modify-Write cycles=0A= - That translates to roughly to 6*140=3D840 I/Os per second.=0A= - The 14 SATA disks offer a maximum of 14*90=3D1260 I/Os per second=0A= - No wonder that the NFS server gives usually a 20% wait I/O usage.=0A= =0A= =0A= =0A= We spearated that stricly. Otherwise we would go with a Gluster FS.=0A= =0A= Markus=0A= =0A= P.S. We leave VMWare because live storage migration of OVirt VMs =0A= gives us the possibility to reduce the complexity on the storage side.=0A= ------=_NextPartTM-000-dc553fe8-e084-4205-a5f2-e4e48c40f662 Content-Type: text/plain; name="InterScan_Disclaimer.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="InterScan_Disclaimer.txt" **************************************************************************** Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Über das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 **************************************************************************** ------=_NextPartTM-000-dc553fe8-e084-4205-a5f2-e4e48c40f662--

better go for iscsi or something else... i whould avoid nfs for vm hosting Freebsd10 delivers kernel iscsitarget now, which works great so far. or go with omnios to get comstar iscsi, which is a rocksolid solution Cheers, Juergen On Wed, Jan 8, 2014 at 2:34 PM, noc <noc(a)nieuwland.nl&gt; wrote: -- Sent from the Delta quadrant using Borg technology!

On Thu, 2014-01-09 at 07:10 +0000, Markus Stockhausen wrote: Only technical difference I can think of is the iSCSI-level load-balancing. With NFS you set up the network with LACP and let that load-balance for you (and you should probably do that with iSCSI as well but you don´t strictly have to). I think it has to do with a chance of trying to go beyond the capacity of 1 network interface at the same time, from one Host (higher bandwidth) that makes people try iSCSI instead of plain NFS. I have tried that but was never able to achieve that effect, so in our situation, there´s no difference. In comparing them both in benchmarks, there was no performance difference at all, at least for our storage systems that are based on FreeBSD. /K

On Thu, 2014-01-09 at 08:35 +0100, squadra wrote: Yes, that´s what I meant, maybe could have been more clear about that, sorry. Multipathing (and the load-balancing it brings) is what really separates iSCSI from NFS. What I´d be interested in knowing is at what breaking-point, not having multipathing becomes an issue. I mean, we might not have such a big VM-park, about 300-400 VMs. But so far running without multipathing using good ole' NFS and no performance issues this far. Would be good to know beforehand if we´re headed for a wall of some sorts, and about "when" we´ll hit it... /K

This is a multi-part message in MIME format. ------=_NextPartTM-000-137c6f94-d5f4-4972-a21f-96e342f08996 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable =0A= =0A= =0A= =0A= =0A= If that is really a concern for the initial question about a "low cost NFS= =0A= solution" LACP on the NFS filer side will mitigate the bottleneck from =0A= too many hypervisors. =0A= =0A= My personal headache is the I/O performance of QEMU. More details here:=0A= http://lists.nongnu.org/archive/html/qemu-discuss/2013-12/msg00028.html=0A= Or to make it short: Each I/O in a VM gets a penalty of 370us. That is much= =0A= more than in ESX environments.=0A= =0A= I would be interested if this the same in ISCSI setups.=0A= =0A= Markus=0A= ------=_NextPartTM-000-137c6f94-d5f4-4972-a21f-96e342f08996 Content-Type: text/plain; name="InterScan_Disclaimer.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="InterScan_Disclaimer.txt" **************************************************************************** Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Über das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 **************************************************************************** ------=_NextPartTM-000-137c6f94-d5f4-4972-a21f-96e342f08996--

On Thu, 2014-01-09 at 09:30 +0100, squadra wrote: Yeah I see 10´s reached RC4 now, probably´ll be out for real soon, and then a while more to wait for 10.1 to have longer support:) Have you compared the new iscsi-target with "ports/istgt" btw? /K

On Thu, Jan 9, 2014 at 9:39 AM, Markus Stockhausen <stockhausen(a)collogia.de&gt; wrote: There are some interesting tests here: http://www.linux-kvm.org/page/Virtio/Block/Latency Results seem to depend a lot on the guest OS IO stack/drivers (I see you use win2k3?).

There's are already enaugh articles on the web about NFS problems related locking, latency, etc.... Eh stacking a protocol onto another to fix problem and then maybe one more to glue them together. Google for the suse PDF " why NFS sucks", I don't agree with the whole sheet.. NFS got his place,too. But not as production filer for VM. Cheers, Juergen, the NFS lover On Jan 9, 2014 8:10 AM, "Markus Stockhausen" <stockhausen(a)collogia.de&gt; wrote: