Re: [Users] oVirt and Infiniband

On Tue, 2013-08-20 at 00:24 +0200, René Koch wrote: > > -----Original message----- >> From:Dan Kenigsberg <danken(a)redhat.com&gt; >> Sent: Monday 19th August 2013 23:48 >> To: René Koch <r.koch(a)ovido.at>; Itzik Brown <ItzikB(a)mellanox.com&gt; >> Cc: ovirt-users <users(a)ovirt.org&gt; >> Subject: Re: [Users] oVirt and Infiniband >> >> On Tue, Aug 13, 2013 at 03:48:14PM +0200, René Koch (ovido) wrote: >>> Hi, >>> >>> Does anyone have oVirt (or a plain KVM hypervisor) running on Infiniband >>> where the virtual machines are bridged to an Infiniband interface (with >>> EoIB kernel module)? >> >> As far as I recall the EoIB module is not yet in upstream kernel. Could >> you give more details on your setup (distro, kernel, module)? >> Do you get connectivity between the hosts? Maybe Itzik and his >> colleagues could help us here. > > > Thanks for your answer. > > That's right - EoIB isn't in the kernel. > I installed module from Mellanox OFED package and even there it's marked as unstable... > OS is CentOS 6.4. > > At the moment I've also an open support case with Mellanox trying to solve the connectivity issue. > So I have support from official site, but was thinking maybe someone has already experience with such a setup and also had some issues/tipps for me... > > My problem is some sort of a bridge or maybe MAC translation issue (unsure cause of a lot of unanswered ARP requests in tcpdump and bridge is working on Ethernet interfaces). > I have bridge ovirtmgmt on eth2 interface (eth2 is a (virtual) ethernet interface bound to the Infiniband-interface) with an IP address configured on it. > Hosts can reach each other, so ethernet to infiniband translation seems to work. > But when I create a vm in oVirt this vm can only communicate with IPs/vms on the bridge. > This mean I can ping the IP of the host and other vms on this bridge, but no host behind the bridge (like e.g. other hosts or vms on other hosts)... > Mellanox support team found the issue: Daemon openibd requires write access to libvirt which is restricted by oVirt per default. When changing auth_unix_rw to "none" networking of the vms is working fine over the inifinband network. So my question is now: I think there's a good reason why write access to libvirt is restricted. In my particular setup no one will do a virsh start/stop/whatever so from a user point I can live with an open libvirt. But are there any troubles I can run into from oVirt side with auth_unix_rw="none" beside users doing evil virsh stuff? Today I tested it and ran into a first issue which I didn't investigate so far (will do this week): - Started vm from oVirt on node - Changed auth_unix_rw to "none" - Restarted libvirt on node - VM was running according to "virsh list" - VM was stopped in oVirt - Started vm in oVirt on another node - vm was running twice Will have a look at the logs what append exactly this vm...

> > Regards, > René > > >> >>> >>> I'm having issues in such a setup where vms can't communicate over this >>> bridge... >>> >>> >>> Regards, >>> René >>> >>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >> > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users