[Users] oVirt and Infiniband

[René Koch (ovido)]

On Tue, 2013-08-20 at 00:24 +0200, René Koch wrote:
>  
> -----Original message-----
> > From:Dan Kenigsberg <danken at redhat.com>
> > Sent: Monday 19th August 2013 23:48
> > To: René Koch <r.koch at ovido.at>; Itzik Brown <ItzikB at mellanox.com>
> > Cc: ovirt-users <users at ovirt.org>
> > Subject: Re: [Users] oVirt and Infiniband
> > 
> > On Tue, Aug 13, 2013 at 03:48:14PM +0200, René Koch (ovido) wrote:
> > > Hi,
> > > 
> > > Does anyone have oVirt (or a plain KVM hypervisor) running on Infiniband
> > > where the virtual machines are bridged to an Infiniband interface (with
> > > EoIB kernel module)?
> > 
> > As far as I recall the EoIB module is not yet in upstream kernel. Could
> > you give more details on your setup (distro, kernel, module)?
> > Do you get connectivity between the hosts? Maybe Itzik and his
> > colleagues could help us here.
> 
> 
> Thanks for your answer.
> 
> That's right - EoIB isn't in the kernel.
> I installed module from Mellanox OFED package and even there it's marked as unstable...
> OS is CentOS 6.4.
> 
> At the moment I've also an open support case with Mellanox trying to solve the connectivity issue.
> So I have support from official site, but was thinking maybe someone has already experience with such a setup and also had some issues/tipps for me...
> 
> My problem is some sort of a bridge or maybe MAC translation issue (unsure cause of a lot of unanswered ARP requests in tcpdump and bridge is working on Ethernet interfaces).
> I have bridge ovirtmgmt on eth2 interface (eth2 is a (virtual) ethernet interface bound to the Infiniband-interface) with an IP address configured on it.
> Hosts can reach each other, so ethernet to infiniband translation seems to work.
> But when I create a vm in oVirt this vm can only communicate with IPs/vms on the bridge.
> This mean I can ping the IP of the host and other vms on this bridge, but no host behind the bridge (like e.g. other hosts or vms on other hosts)...
> 


Mellanox support team found the issue:
Daemon openibd requires write access to libvirt which is restricted by
oVirt per default.

When changing auth_unix_rw to "none" networking of the vms is working fine
over the inifinband network.

So my question is now:
I think there's a good reason why write access to libvirt is restricted.
In my particular setup no one will do a virsh start/stop/whatever so
from a user point I can live with an open libvirt.

But are there any troubles I can run into from oVirt side with
auth_unix_rw="none" beside users doing evil virsh stuff?

Today I tested it and ran into a first issue which I didn't investigate
so far (will do this week):
- Started vm from oVirt on node
- Changed auth_unix_rw to "none"
- Restarted libvirt on node
- VM was running according to "virsh list"
- VM was stopped in oVirt
- Started vm in oVirt on another node
- vm was running twice
Will have a look at the logs what append exactly this vm...


> 
> Regards,
> René
> 
> 
> > 
> > > 
> > > I'm having issues in such a setup where vms can't communicate over this
> > > bridge...
> > > 
> > > 
> > > Regards,
> > > René
> > > 
> > > 
> > > 
> > > _______________________________________________
> > > Users mailing list
> > > Users at ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/users
> > 
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users