[Users] oVirt and Infiniband

[Itamar Heim]

On 08/21/2013 03:32 AM, René Koch (ovido) wrote:
>
> On Tue, 2013-08-20 at 00:24 +0200, René Koch wrote:
>>
>> -----Original message-----
>>> From:Dan Kenigsberg <danken at redhat.com>
>>> Sent: Monday 19th August 2013 23:48
>>> To: René Koch <r.koch at ovido.at>; Itzik Brown <ItzikB at mellanox.com>
>>> Cc: ovirt-users <users at ovirt.org>
>>> Subject: Re: [Users] oVirt and Infiniband
>>>
>>> On Tue, Aug 13, 2013 at 03:48:14PM +0200, René Koch (ovido) wrote:
>>>> Hi,
>>>>
>>>> Does anyone have oVirt (or a plain KVM hypervisor) running on Infiniband
>>>> where the virtual machines are bridged to an Infiniband interface (with
>>>> EoIB kernel module)?
>>>
>>> As far as I recall the EoIB module is not yet in upstream kernel. Could
>>> you give more details on your setup (distro, kernel, module)?
>>> Do you get connectivity between the hosts? Maybe Itzik and his
>>> colleagues could help us here.
>>
>>
>> Thanks for your answer.
>>
>> That's right - EoIB isn't in the kernel.
>> I installed module from Mellanox OFED package and even there it's marked as unstable...
>> OS is CentOS 6.4.
>>
>> At the moment I've also an open support case with Mellanox trying to solve the connectivity issue.
>> So I have support from official site, but was thinking maybe someone has already experience with such a setup and also had some issues/tipps for me...
>>
>> My problem is some sort of a bridge or maybe MAC translation issue (unsure cause of a lot of unanswered ARP requests in tcpdump and bridge is working on Ethernet interfaces).
>> I have bridge ovirtmgmt on eth2 interface (eth2 is a (virtual) ethernet interface bound to the Infiniband-interface) with an IP address configured on it.
>> Hosts can reach each other, so ethernet to infiniband translation seems to work.
>> But when I create a vm in oVirt this vm can only communicate with IPs/vms on the bridge.
>> This mean I can ping the IP of the host and other vms on this bridge, but no host behind the bridge (like e.g. other hosts or vms on other hosts)...
>>
>
>
> Mellanox support team found the issue:
> Daemon openibd requires write access to libvirt which is restricted by
> oVirt per default.
>
> When changing auth_unix_rw to "none" networking of the vms is working fine
> over the inifinband network.
>
> So my question is now:
> I think there's a good reason why write access to libvirt is restricted.
> In my particular setup no one will do a virsh start/stop/whatever so
> from a user point I can live with an open libvirt.
>
> But are there any troubles I can run into from oVirt side with
> auth_unix_rw="none" beside users doing evil virsh stuff?
>
> Today I tested it and ran into a first issue which I didn't investigate
> so far (will do this week):
> - Started vm from oVirt on node
> - Changed auth_unix_rw to "none"
> - Restarted libvirt on node
> - VM was running according to "virsh list"
> - VM was stopped in oVirt
> - Started vm in oVirt on another node
> - vm was running twice
> Will have a look at the logs what append exactly this vm...
>

isn't the right thing is to configure openibd to use same credentials 
vdsm is using (or with their own credentials)?

>
>>
>> Regards,
>> René
>>
>>
>>>
>>>>
>>>> I'm having issues in such a setup where vms can't communicate over this
>>>> bridge...
>>>>
>>>>
>>>> Regards,
>>>> René
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>