Hi, Does anyone have oVirt (or a plain KVM hypervisor) running on Infiniband where the virtual machines are bridged to an Infiniband interface (with EoIB kernel module)?

I'm having issues in such a setup where vms can't communicate over this bridge... Regards, René _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

-----Original message----- > From:Dan Kenigsberg <danken(a)redhat.com&gt; > Sent: Monday 19th August 2013 23:48 > To: René Koch <r.koch(a)ovido.at>; Itzik Brown <ItzikB(a)mellanox.com&gt; > Cc: ovirt-users <users(a)ovirt.org&gt; > Subject: Re: [Users] oVirt and Infiniband > > On Tue, Aug 13, 2013 at 03:48:14PM +0200, René Koch (ovido) wrote: > > Hi, > > > > Does anyone have oVirt (or a plain KVM hypervisor) running on Infiniband > > where the virtual machines are bridged to an Infiniband interface (with > > EoIB kernel module)? > > As far as I recall the EoIB module is not yet in upstream kernel. Could > you give more details on your setup (distro, kernel, module)? > Do you get connectivity between the hosts? Maybe Itzik and his > colleagues could help us here. Thanks for your answer. That's right - EoIB isn't in the kernel. I installed module from Mellanox OFED package and even there it's marked as unstable... OS is CentOS 6.4. At the moment I've also an open support case with Mellanox trying to solve the connectivity issue. So I have support from official site, but was thinking maybe someone has already experience with such a setup and also had some issues/tipps for me... My problem is some sort of a bridge or maybe MAC translation issue (unsure cause of a lot of unanswered ARP requests in tcpdump and bridge is working on Ethernet interfaces). I have bridge ovirtmgmt on eth2 interface (eth2 is a (virtual) ethernet interface bound to the Infiniband-interface) with an IP address configured on it. Hosts can reach each other, so ethernet to infiniband translation seems to work. But when I create a vm in oVirt this vm can only communicate with IPs/vms on the bridge. This mean I can ping the IP of the host and other vms on this bridge, but no host behind the bridge (like e.g. other hosts or vms on other hosts)...

Regards, René > > > > > I'm having issues in such a setup where vms can't communicate over this > > bridge... > > > > > > Regards, > > René > > > > > > > > _______________________________________________ > > Users mailing list > > Users(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/users > _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

On 08/21/2013 03:32 AM, René Koch (ovido) wrote: > > On Tue, 2013-08-20 at 00:24 +0200, René Koch wrote: >> >> -----Original message----- >>> From:Dan Kenigsberg <danken(a)redhat.com&gt; >>> Sent: Monday 19th August 2013 23:48 >>> To: René Koch <r.koch(a)ovido.at>; Itzik Brown <ItzikB(a)mellanox.com&gt; >>> Cc: ovirt-users <users(a)ovirt.org&gt; >>> Subject: Re: [Users] oVirt and Infiniband >>> >>> On Tue, Aug 13, 2013 at 03:48:14PM +0200, René Koch (ovido) wrote: >>>> Hi, >>>> >>>> Does anyone have oVirt (or a plain KVM hypervisor) running on Infiniband >>>> where the virtual machines are bridged to an Infiniband interface (with >>>> EoIB kernel module)? >>> >>> As far as I recall the EoIB module is not yet in upstream kernel. Could >>> you give more details on your setup (distro, kernel, module)? >>> Do you get connectivity between the hosts? Maybe Itzik and his >>> colleagues could help us here. >> >> >> Thanks for your answer. >> >> That's right - EoIB isn't in the kernel. >> I installed module from Mellanox OFED package and even there it's marked as unstable... >> OS is CentOS 6.4. >> >> At the moment I've also an open support case with Mellanox trying to solve the connectivity issue. >> So I have support from official site, but was thinking maybe someone has already experience with such a setup and also had some issues/tipps for me... >> >> My problem is some sort of a bridge or maybe MAC translation issue (unsure cause of a lot of unanswered ARP requests in tcpdump and bridge is working on Ethernet interfaces). >> I have bridge ovirtmgmt on eth2 interface (eth2 is a (virtual) ethernet interface bound to the Infiniband-interface) with an IP address configured on it. >> Hosts can reach each other, so ethernet to infiniband translation seems to work. >> But when I create a vm in oVirt this vm can only communicate with IPs/vms on the bridge. >> This mean I can ping the IP of the host and other vms on this bridge, but no host behind the bridge (like e.g. other hosts or vms on other hosts)... >> > > > Mellanox support team found the issue: > Daemon openibd requires write access to libvirt which is restricted by > oVirt per default. > > When changing auth_unix_rw to "none" networking of the vms is working fine > over the inifinband network. > > So my question is now: > I think there's a good reason why write access to libvirt is restricted. > In my particular setup no one will do a virsh start/stop/whatever so > from a user point I can live with an open libvirt. > > But are there any troubles I can run into from oVirt side with > auth_unix_rw="none" beside users doing evil virsh stuff? > > Today I tested it and ran into a first issue which I didn't investigate > so far (will do this week): > - Started vm from oVirt on node > - Changed auth_unix_rw to "none" > - Restarted libvirt on node > - VM was running according to "virsh list" > - VM was stopped in oVirt > - Started vm in oVirt on another node > - vm was running twice > Will have a look at the logs what append exactly this vm... > isn't the right thing is to configure openibd to use same credentials vdsm is using (or with their own credentials)?

> >> >> Regards, >> René >> >> >>> >>>> >>>> I'm having issues in such a setup where vms can't communicate over this >>>> bridge... >>>> >>>> >>>> Regards, >>>> René >>>> >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >