[Users] oVirt and Infiniband
Hi, Does anyone have oVirt (or a plain KVM hypervisor) running on Infiniband where the virtual machines are bridged to an Infiniband interface (with EoIB kernel module)? I'm having issues in such a setup where vms can't communicate over this bridge... Regards, René
On Tue, Aug 13, 2013 at 03:48:14PM +0200, René Koch (ovido) wrote:
Hi,
Does anyone have oVirt (or a plain KVM hypervisor) running on Infiniband where the virtual machines are bridged to an Infiniband interface (with EoIB kernel module)?
As far as I recall the EoIB module is not yet in upstream kernel. Could you give more details on your setup (distro, kernel, module)? Do you get connectivity between the hosts? Maybe Itzik and his colleagues could help us here.
I'm having issues in such a setup where vms can't communicate over this bridge...
Regards, René
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-----Original message-----
From:Dan Kenigsberg <danken@redhat.com> Sent: Monday 19th August 2013 23:48 To: René Koch <r.koch@ovido.at>; Itzik Brown <ItzikB@mellanox.com> Cc: ovirt-users <users@ovirt.org> Subject: Re: [Users] oVirt and Infiniband
On Tue, Aug 13, 2013 at 03:48:14PM +0200, René Koch (ovido) wrote:
Hi,
Does anyone have oVirt (or a plain KVM hypervisor) running on Infiniband where the virtual machines are bridged to an Infiniband interface (with EoIB kernel module)?
As far as I recall the EoIB module is not yet in upstream kernel. Could you give more details on your setup (distro, kernel, module)? Do you get connectivity between the hosts? Maybe Itzik and his colleagues could help us here.
Thanks for your answer. That's right - EoIB isn't in the kernel. I installed module from Mellanox OFED package and even there it's marked as unstable... OS is CentOS 6.4. At the moment I've also an open support case with Mellanox trying to solve the connectivity issue. So I have support from official site, but was thinking maybe someone has already experience with such a setup and also had some issues/tipps for me... My problem is some sort of a bridge or maybe MAC translation issue (unsure cause of a lot of unanswered ARP requests in tcpdump and bridge is working on Ethernet interfaces). I have bridge ovirtmgmt on eth2 interface (eth2 is a (virtual) ethernet interface bound to the Infiniband-interface) with an IP address configured on it. Hosts can reach each other, so ethernet to infiniband translation seems to work. But when I create a vm in oVirt this vm can only communicate with IPs/vms on the bridge. This mean I can ping the IP of the host and other vms on this bridge, but no host behind the bridge (like e.g. other hosts or vms on other hosts)... Regards, René
I'm having issues in such a setup where vms can't communicate over this bridge...
Regards, René
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Tue, 2013-08-20 at 00:24 +0200, René Koch wrote:
-----Original message-----
From:Dan Kenigsberg <danken@redhat.com> Sent: Monday 19th August 2013 23:48 To: René Koch <r.koch@ovido.at>; Itzik Brown <ItzikB@mellanox.com> Cc: ovirt-users <users@ovirt.org> Subject: Re: [Users] oVirt and Infiniband
On Tue, Aug 13, 2013 at 03:48:14PM +0200, René Koch (ovido) wrote:
Hi,
Does anyone have oVirt (or a plain KVM hypervisor) running on Infiniband where the virtual machines are bridged to an Infiniband interface (with EoIB kernel module)?
As far as I recall the EoIB module is not yet in upstream kernel. Could you give more details on your setup (distro, kernel, module)? Do you get connectivity between the hosts? Maybe Itzik and his colleagues could help us here.
Thanks for your answer.
That's right - EoIB isn't in the kernel. I installed module from Mellanox OFED package and even there it's marked as unstable... OS is CentOS 6.4.
At the moment I've also an open support case with Mellanox trying to solve the connectivity issue. So I have support from official site, but was thinking maybe someone has already experience with such a setup and also had some issues/tipps for me...
My problem is some sort of a bridge or maybe MAC translation issue (unsure cause of a lot of unanswered ARP requests in tcpdump and bridge is working on Ethernet interfaces). I have bridge ovirtmgmt on eth2 interface (eth2 is a (virtual) ethernet interface bound to the Infiniband-interface) with an IP address configured on it. Hosts can reach each other, so ethernet to infiniband translation seems to work. But when I create a vm in oVirt this vm can only communicate with IPs/vms on the bridge. This mean I can ping the IP of the host and other vms on this bridge, but no host behind the bridge (like e.g. other hosts or vms on other hosts)...
Mellanox support team found the issue: Daemon openibd requires write access to libvirt which is restricted by oVirt per default. When changing auth_unix_rw to "none" networking of the vms is working fine over the inifinband network. So my question is now: I think there's a good reason why write access to libvirt is restricted. In my particular setup no one will do a virsh start/stop/whatever so from a user point I can live with an open libvirt. But are there any troubles I can run into from oVirt side with auth_unix_rw="none" beside users doing evil virsh stuff? Today I tested it and ran into a first issue which I didn't investigate so far (will do this week): - Started vm from oVirt on node - Changed auth_unix_rw to "none" - Restarted libvirt on node - VM was running according to "virsh list" - VM was stopped in oVirt - Started vm in oVirt on another node - vm was running twice Will have a look at the logs what append exactly this vm...
Regards, René
I'm having issues in such a setup where vms can't communicate over this bridge...
Regards, René
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On 08/21/2013 03:32 AM, René Koch (ovido) wrote:
On Tue, 2013-08-20 at 00:24 +0200, René Koch wrote:
-----Original message-----
From:Dan Kenigsberg <danken@redhat.com> Sent: Monday 19th August 2013 23:48 To: René Koch <r.koch@ovido.at>; Itzik Brown <ItzikB@mellanox.com> Cc: ovirt-users <users@ovirt.org> Subject: Re: [Users] oVirt and Infiniband
On Tue, Aug 13, 2013 at 03:48:14PM +0200, René Koch (ovido) wrote:
Hi,
Does anyone have oVirt (or a plain KVM hypervisor) running on Infiniband where the virtual machines are bridged to an Infiniband interface (with EoIB kernel module)?
As far as I recall the EoIB module is not yet in upstream kernel. Could you give more details on your setup (distro, kernel, module)? Do you get connectivity between the hosts? Maybe Itzik and his colleagues could help us here.
Thanks for your answer.
That's right - EoIB isn't in the kernel. I installed module from Mellanox OFED package and even there it's marked as unstable... OS is CentOS 6.4.
At the moment I've also an open support case with Mellanox trying to solve the connectivity issue. So I have support from official site, but was thinking maybe someone has already experience with such a setup and also had some issues/tipps for me...
My problem is some sort of a bridge or maybe MAC translation issue (unsure cause of a lot of unanswered ARP requests in tcpdump and bridge is working on Ethernet interfaces). I have bridge ovirtmgmt on eth2 interface (eth2 is a (virtual) ethernet interface bound to the Infiniband-interface) with an IP address configured on it. Hosts can reach each other, so ethernet to infiniband translation seems to work. But when I create a vm in oVirt this vm can only communicate with IPs/vms on the bridge. This mean I can ping the IP of the host and other vms on this bridge, but no host behind the bridge (like e.g. other hosts or vms on other hosts)...
Mellanox support team found the issue: Daemon openibd requires write access to libvirt which is restricted by oVirt per default.
When changing auth_unix_rw to "none" networking of the vms is working fine over the inifinband network.
So my question is now: I think there's a good reason why write access to libvirt is restricted. In my particular setup no one will do a virsh start/stop/whatever so from a user point I can live with an open libvirt.
But are there any troubles I can run into from oVirt side with auth_unix_rw="none" beside users doing evil virsh stuff?
Today I tested it and ran into a first issue which I didn't investigate so far (will do this week): - Started vm from oVirt on node - Changed auth_unix_rw to "none" - Restarted libvirt on node - VM was running according to "virsh list" - VM was stopped in oVirt - Started vm in oVirt on another node - vm was running twice Will have a look at the logs what append exactly this vm...
isn't the right thing is to configure openibd to use same credentials vdsm is using (or with their own credentials)?
Regards, René
I'm having issues in such a setup where vms can't communicate over this bridge...
Regards, René
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Thu, 2013-08-22 at 05:16 -0400, Itamar Heim wrote:
On 08/21/2013 03:32 AM, René Koch (ovido) wrote:
On Tue, 2013-08-20 at 00:24 +0200, René Koch wrote:
-----Original message-----
From:Dan Kenigsberg <danken@redhat.com> Sent: Monday 19th August 2013 23:48 To: René Koch <r.koch@ovido.at>; Itzik Brown <ItzikB@mellanox.com> Cc: ovirt-users <users@ovirt.org> Subject: Re: [Users] oVirt and Infiniband
On Tue, Aug 13, 2013 at 03:48:14PM +0200, René Koch (ovido) wrote:
Hi,
Does anyone have oVirt (or a plain KVM hypervisor) running on Infiniband where the virtual machines are bridged to an Infiniband interface (with EoIB kernel module)?
As far as I recall the EoIB module is not yet in upstream kernel. Could you give more details on your setup (distro, kernel, module)? Do you get connectivity between the hosts? Maybe Itzik and his colleagues could help us here.
Thanks for your answer.
That's right - EoIB isn't in the kernel. I installed module from Mellanox OFED package and even there it's marked as unstable... OS is CentOS 6.4.
At the moment I've also an open support case with Mellanox trying to solve the connectivity issue. So I have support from official site, but was thinking maybe someone has already experience with such a setup and also had some issues/tipps for me...
My problem is some sort of a bridge or maybe MAC translation issue (unsure cause of a lot of unanswered ARP requests in tcpdump and bridge is working on Ethernet interfaces). I have bridge ovirtmgmt on eth2 interface (eth2 is a (virtual) ethernet interface bound to the Infiniband-interface) with an IP address configured on it. Hosts can reach each other, so ethernet to infiniband translation seems to work. But when I create a vm in oVirt this vm can only communicate with IPs/vms on the bridge. This mean I can ping the IP of the host and other vms on this bridge, but no host behind the bridge (like e.g. other hosts or vms on other hosts)...
Mellanox support team found the issue: Daemon openibd requires write access to libvirt which is restricted by oVirt per default.
When changing auth_unix_rw to "none" networking of the vms is working fine over the inifinband network.
So my question is now: I think there's a good reason why write access to libvirt is restricted. In my particular setup no one will do a virsh start/stop/whatever so from a user point I can live with an open libvirt.
But are there any troubles I can run into from oVirt side with auth_unix_rw="none" beside users doing evil virsh stuff?
Today I tested it and ran into a first issue which I didn't investigate so far (will do this week): - Started vm from oVirt on node - Changed auth_unix_rw to "none" - Restarted libvirt on node - VM was running according to "virsh list" - VM was stopped in oVirt - Started vm in oVirt on another node - vm was running twice Will have a look at the logs what append exactly this vm...
isn't the right thing is to configure openibd to use same credentials vdsm is using (or with their own credentials)?
Absolutely agree that this would be the best solution, but unfortunately there's no config option for openibd to use credentials when connecting to libvirt :(
Regards, René
I'm having issues in such a setup where vms can't communicate over this bridge...
Regards, René
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (4)
-
Dan Kenigsberg -
Itamar Heim -
René Koch -
René Koch (ovido)