This is a multi-part message in MIME format.

------=_NextPartTM-000-55ab7768-e2e6-4bb1-a471-e6535eb723ff
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

> Von: users-bounces@ovirt.org [users-bounces@ovirt.org]" im Auftrag v=
on "Ren=E9 Koch [rkoch@linuxland.at]=0A=
> Gesendet: Freitag, 30. Mai 2014 21:08=0A=
> An: ml ml=0A=
> Cc: users@ovirt.org=0A=
> Betreff: Re: [ovirt-users] oVirt Infiniband Support - Etherrnet vs Infini=
band=0A=
> =0A=
> On 05/30/2014 04:55 PM, ml ml wrote:=0A=
> > Hello List,=0A=
> >=0A=
> > i am very new to ovirt. I am setting up a 2 Node Cluster with GlusterFS=
=0A=
> > Sync Replication.=0A=
> >=0A=
> > I was about to buy a 2-Port Intel Gbit nic for each host which is about=
=0A=
> > 150Eur each.=0A=
> >=0A=
> > However, for a little extra of 180Eur you get a 10GBit Infiniband -=0A=
> > MCX311A-XCAT.=0A=
> >=0A=
> > I would like to have quick opinions on this.=0A=
> >=0A=
> > Should i pay a little more and then get 10MBit instead of 2GBit bonded?=
=0A=
> =0A=
> Infiniband is good for GlusterFS / storage connection but never never=0A=
> never never ever use it as a vm network - believe me it's a nightmare.=0A=
> =0A=
> So you would direct connect your 2 nodes as otherwise you would have to=
=0A=
> buy an Infiniband switch as well (btw I'm not sure if direct connect is=
=0A=
> possible, but I guess it is). Keep in mind that Infiniband cables aren't=
=0A=
> cheap, too.=0A=
> Why don't you go for 10Gbit/s Ethernet?=0A=
=0A=
Welcome to Ovirt.=0A=
=0A=
We are on infiniband IPoIB for about 3 years now (started with VMWare). So =
we=0A=
jumped on the 10GBit path long before 10GB ethernet had reasonable prices. =
=0A=
We are fine with it but you should consider the following aspects.=0A=
=0A=
- Go for ConnectX-2 cards or higher. They allow for IPoIB TCP offloading. D=
o not=0A=
believe the other information you will find on the internet. A helful statm=
ent here:=0A=
http://permalink.gmane.org/gmane.linux.drivers.rdma/18032=0A=
=0A=
- With IPoIB infiniband you should always be happy to achieve more than 1GB=
it =0A=
throughput. A hear a lot of people complaining that they do not get wire sp=
eed.=0A=
The stack is quite complex and from our experience we can reach 5-7GBit spe=
eds =0A=
easily on a DDR backbone (16 Gbps payload). Better CPUs will increase the =
=0A=
throughput. A jump from Xeon 5400 to 5600 gave a nice boost.=0A=
=0A=
- Everything around NFS over RDMA is currently a mess. =0A=
=0A=
- Cable prices are a real pain (at least in Europe). =0A=
=0A=
- Older CX4 Switches are very cheap and rock solid. We have a CISCO 7000D.=
=0A=
Though limited to 2044MTU they are fine.=0A=
=0A=
- Infiniband requires a subnet manager on the network. Cards will only get =
a=0A=
link if that process is available. Either you need a switch with an builtin=
 SM or=0A=
in case of direct connection it must run on one of the hosts.=0A=
=0A=
- always remember that GBit bonding will not improve the throughput of one=
=0A=
connection. So you can easily migrate a single VM or disks over infiniband =
with =0A=
much more than 1GBit while an ethernet bond will be limited to 1Gbit.=0A=
=0A=
Maybe you should have a look at the 10GBit mellanox ethernet adapters. They=
 are =0A=
floating around for cheap prices ~$60 have a SFP+ port and are build from t=
he=0A=
same silicone as the infiniband cards. For direct connections they should b=
e fine=0A=
but switching hardware is expensive.=0A=
=0A=
Markus=0A=
------=_NextPartTM-000-55ab7768-e2e6-4bb1-a471-e6535eb723ff
Content-Type: text/plain;
	name="InterScan_Disclaimer.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename="InterScan_Disclaimer.txt"

****************************************************************************
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail ist nicht gestattet.

Über das Internet versandte E-Mails können unter fremden Namen erstellt oder
manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine
rechtsverbindliche Willenserklärung.

Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln

Vorstand:
Kadir Akin
Dr. Michael Höhnerbach

Vorsitzender des Aufsichtsrates:
Hans Kristian Langva

Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497

This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.

e-mails sent over the internet may have been written under a wrong name or
been manipulated. That is why this message sent as an e-mail is not a
legally binding declaration of intention.

Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln

executive board:
Kadir Akin
Dr. Michael Höhnerbach

President of the supervisory board:
Hans Kristian Langva

Registry office: district court Cologne
Register number: HRB 52 497

****************************************************************************

------=_NextPartTM-000-55ab7768-e2e6-4bb1-a471-e6535eb723ff--