6:43 a.m.
Hi list,
I'm trying a test installation of the new Ovirt Node Image and engine before
upgrading my 3.6 installation.
In my current environment every hypervisor access the NFS shared data domain via
infiniband. With a couple of vdsm hooks I was able to set the infiniband link in
connected mode and set a MTU of 65520. In my case the infiniband link is used as
a common NIC, I don't use RDMA, only IP over infiniband to access the data domain.
Now, when I attempt to setup host networking, the system try to bridge the
connection "ibstorage" (defined in networks tab) with ib0 device. But a non
ethernet device cannot be bridged that way and host networking setup fails.
Any idea or suggestion? With 3.6 (node and engine) all was fine.
Thanks
--
gb
PGP Key: http://pgp.mit.edu/
Primary key fingerprint: C510 0765 943E EBED A4F2 69D3 16CC DC90 B9CB 0F34
8:58 a.m.
Hello,
What's "couple of vdsm hooks"? Could you provide the source code of those?
Regards,
Yevgeny
On Mon, Jul 4, 2016 at 1:43 PM, Giorgio Biacchi <giorgio(a)di.unimi.it> wrote:
Hi list,
I'm trying a test installation of the new Ovirt Node Image and engine
before
upgrading my 3.6 installation.
In my current environment every hypervisor access the NFS shared data
domain via
infiniband. With a couple of vdsm hooks I was able to set the infiniband
link in
connected mode and set a MTU of 65520. In my case the infiniband link is
used as
a common NIC, I don't use RDMA, only IP over infiniband to access the data
domain.
Now, when I attempt to setup host networking, the system try to bridge the
connection "ibstorage" (defined in networks tab) with ib0 device. But a non
ethernet device cannot be bridged that way and host networking setup fails.
Any idea or suggestion? With 3.6 (node and engine) all was fine.
Thanks
--
gb
PGP Key: http://pgp.mit.edu/
Primary key fingerprint: C510 0765 943E EBED A4F2 69D3 16CC DC90 B9CB 0F34
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
9:24 a.m.
Hi,
hooks are only needed to set connected mode and the correct MTU on the
infiniband link. The problem is bound to the host networking setup procedure.
Here's the hooks:
cat /usr/libexec/vdsm/hooks/before_vdsm_start/10_set_ib0_connected_mode
#!/bin/sh
echo Setting IB connected mode
echo connected > /sys/class/net/ib0/mode
sleep 3
MODE=$(cat /sys/class/net/ib0/mode)
RATE=$(cat /sys/class/infiniband/mlx4_0/ports/1/rate)
echo ib0 is now in $MODE mode with rate $RATE
cat /usr/libexec/vdsm/hooks/after_network_setup/10_set_ib0_connected_mode
#!/bin/sh
echo connected > /sys/class/net/ib0/mode
sleep 3
ip link set mtu 65520 dev ib0
As you can see they work:
[root@node]# systemctl restart vdsmd
[root@node]# ip l sh dev ib0
5: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc pfifo_fast state UP
mode DEFAULT qlen 256
link/infiniband 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:0a:08:c7
brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
[root@node]# cat /sys/class/net/ib0/mode
connected
In networks tab in the admin portal I've defined a logical network ibstorage
with the correct MTU.
When I attempt to setup host networking vdsm generates the ibstorage device and
a config to bridge it with ib0 but ib0 is not an ethernet device so it cannot be
bridged. With 3.6 the configuration was directly applied to ib0 without any bridge.
Thanks
On 07/04/2016 02:58 PM, Yevgeny Zaspitsky wrote:
Hello,
What's "couple of vdsm hooks"? Could you provide the source code of those?
Regards,
Yevgeny
On Mon, Jul 4, 2016 at 1:43 PM, Giorgio Biacchi <giorgio(a)di.unimi.it
<mailto:giorgio@di.unimi.it>> wrote:
Hi list,
I'm trying a test installation of the new Ovirt Node Image and engine before
upgrading my 3.6 installation.
In my current environment every hypervisor access the NFS shared data domain via
infiniband. With a couple of vdsm hooks I was able to set the infiniband link in
connected mode and set a MTU of 65520. In my case the infiniband link is used as
a common NIC, I don't use RDMA, only IP over infiniband to access the data
domain.
Now, when I attempt to setup host networking, the system try to bridge the
connection "ibstorage" (defined in networks tab) with ib0 device. But a
non
ethernet device cannot be bridged that way and host networking setup fails.
Any idea or suggestion? With 3.6 (node and engine) all was fine.
Thanks
--
gb
PGP Key: http://pgp.mit.edu/
Primary key fingerprint: C510 0765 943E EBED A4F2 69D3 16CC DC90 B9CB 0F34
_______________________________________________
Users mailing list
Users(a)ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
--
gb
PGP Key: http://pgp.mit.edu/
Primary key fingerprint: C510 0765 943E EBED A4F2 69D3 16CC DC90 B9CB 0F34
9:25 a.m.
As I remember the IBoIP is not fully implemented Ethernet protocol, it does
not support bridging.
***********************************************************
Dr. Arman Khalatyan eScience -SuperComputing
Leibniz-Institut für Astrophysik Potsdam (AIP)
An der Sternwarte 16, 14482 Potsdam, Germany
***********************************************************
On Mon, Jul 4, 2016 at 2:58 PM, Yevgeny Zaspitsky <yzaspits(a)redhat.com>
wrote:
Hello,
What's "couple of vdsm hooks"? Could you provide the source code of those?
Regards,
Yevgeny
On Mon, Jul 4, 2016 at 1:43 PM, Giorgio Biacchi <giorgio(a)di.unimi.it>
wrote:
> Hi list,
> I'm trying a test installation of the new Ovirt Node Image and engine
> before
> upgrading my 3.6 installation.
>
> In my current environment every hypervisor access the NFS shared data
> domain via
> infiniband. With a couple of vdsm hooks I was able to set the infiniband
> link in
> connected mode and set a MTU of 65520. In my case the infiniband link is
> used as
> a common NIC, I don't use RDMA, only IP over infiniband to access the
> data domain.
>
> Now, when I attempt to setup host networking, the system try to bridge the
> connection "ibstorage" (defined in networks tab) with ib0 device. But a
> non
> ethernet device cannot be bridged that way and host networking setup
> fails.
>
> Any idea or suggestion? With 3.6 (node and engine) all was fine.
>
> Thanks
> --
> gb
>
> PGP Key: http://pgp.mit.edu/
> Primary key fingerprint: C510 0765 943E EBED A4F2 69D3 16CC DC90 B9CB 0F34
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
10:10 a.m.
As I said, Infiniband is not ethernet so it cannot be L2 bridged. This is normal
and I understand why it can't be done. I'd like to know if there's a way to
configure host networking for infiniband link without bridging because is not
necessary to have a bridge there. In addition to this, 3.6 behavior was
different. Here's all the relevant configuration files generated by 3.6 and 4.0:
------------- 3.6 ---------------
[root(a)node3.6]# cat /etc/sysconfig/network-scripts/ifcfg-ib0
# Generated by VDSM version 4.17.11-7.gitc0752ac.el7
DEVICE=ib0
HWADDR=80:00:00:48:fe:80:00:00:00:00:00:00:00:02:c9:03:00:0a:06:9f
ONBOOT=yes
BOOTPROTO=dhcp
MTU=65520
DEFROUTE=no
NM_CONTROLLED=no
IPV6INIT=no
[root(a)node3.6]# cat /var/lib/vdsm/persistence/netconf/nets/ibstorage
{
"nic": "ib0",
"bootproto": "dhcp",
"blockingdhcp": true,
"bridged": "false",
"mtu": "65520"
}
---------------------------------
------------- 4.0 ---------------
[root(a)node4.0]# cat /etc/sysconfig/network-scripts/ifcfg-ib0
# Generated by VDSM version 4.18.5.1-1.el7.centos
DEVICE=ib0
BRIDGE=ibstorage
ONBOOT=yes
MTU=65520
NM_CONTROLLED=no
IPV6INIT=no
[root(a)node4.0]# cat /var/lib/vdsm/persistence/netconf/nets/ibstorage
{
"ipv6autoconf": false,
"nic": "ib0",
"mtu": 65520,
"switch": "legacy",
"dhcpv6": false,
"stp": false,
"bridged": true,
"defaultRoute": false
}
[root(a)node4.0]# cat /var/lib/vdsm/persistence/netconf/nets/ib0
{
"ipv6autoconf": false,
"nic": "ib0",
"mtu": 65520,
"switch": "legacy",
"dhcpv6": false,
"stp": false,
"bridged": true,
"defaultRoute": false
}
[root(a)node4.0]# cat /etc/sysconfig/network-scripts/ifcfg-ibstorage
# Generated by VDSM version 4.18.5.1-1.el7.centos
DEVICE=ibstorage
TYPE=Bridge
DELAY=0
STP=off
ONBOOT=yes
DEFROUTE=no
NM_CONTROLLED=no
IPV6INIT=no
MTU=65520
---------------------------------
So it seems that 4.0 generates more files and wants to bridge an ethernet
unbridgeable link. Steps to attempt to configure the connection for 4.0 was the
same done successfully for 3.6.
Thanks again
On 07/04/2016 03:25 PM, Arman Khalatyan wrote:
As I remember the IBoIP is not fully implemented Ethernet protocol,
it does not
support bridging.
***********************************************************
Dr. Arman Khalatyan eScience -SuperComputing
Leibniz-Institut für Astrophysik Potsdam (AIP)
An der Sternwarte 16, 14482 Potsdam, Germany
***********************************************************
On Mon, Jul 4, 2016 at 2:58 PM, Yevgeny Zaspitsky <yzaspits(a)redhat.com
<mailto:yzaspits@redhat.com>> wrote:
Hello,
What's "couple of vdsm hooks"? Could you provide the source code of
those?
Regards,
Yevgeny
On Mon, Jul 4, 2016 at 1:43 PM, Giorgio Biacchi <giorgio(a)di.unimi.it
<mailto:giorgio@di.unimi.it>> wrote:
Hi list,
I'm trying a test installation of the new Ovirt Node Image and engine before
upgrading my 3.6 installation.
In my current environment every hypervisor access the NFS shared data
domain via
infiniband. With a couple of vdsm hooks I was able to set the infiniband
link in
connected mode and set a MTU of 65520. In my case the infiniband link is
used as
a common NIC, I don't use RDMA, only IP over infiniband to access the
data domain.
Now, when I attempt to setup host networking, the system try to bridge the
connection "ibstorage" (defined in networks tab) with ib0 device. But a
non
ethernet device cannot be bridged that way and host networking setup fails.
Any idea or suggestion? With 3.6 (node and engine) all was fine.
Thanks
--
gb
PGP Key: http://pgp.mit.edu/
Primary key fingerprint: C510 0765 943E EBED A4F2 69D3 16CC DC90 B9CB 0F34
_______________________________________________
Users mailing list
Users(a)ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users(a)ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
--
gb
PGP Key: http://pgp.mit.edu/
Primary key fingerprint: C510 0765 943E EBED A4F2 69D3 16CC DC90 B9CB 0F34
10:16 a.m.
Infiniband should not be bridged.
Ovirt tries to build a bridge over a NIC if the network is a VM one. Please
verify that it isn't.
On Mon, Jul 4, 2016 at 5:10 PM, Giorgio Biacchi <giorgio(a)di.unimi.it> wrote:
As I said, Infiniband is not ethernet so it cannot be L2 bridged.
This is
normal
and I understand why it can't be done. I'd like to know if there's a way to
configure host networking for infiniband link without bridging because is
not
necessary to have a bridge there. In addition to this, 3.6 behavior was
different. Here's all the relevant configuration files generated by 3.6
and 4.0:
------------- 3.6 ---------------
[root(a)node3.6]# cat /etc/sysconfig/network-scripts/ifcfg-ib0
# Generated by VDSM version 4.17.11-7.gitc0752ac.el7
DEVICE=ib0
HWADDR=80:00:00:48:fe:80:00:00:00:00:00:00:00:02:c9:03:00:0a:06:9f
ONBOOT=yes
BOOTPROTO=dhcp
MTU=65520
DEFROUTE=no
NM_CONTROLLED=no
IPV6INIT=no
[root(a)node3.6]# cat /var/lib/vdsm/persistence/netconf/nets/ibstorage
{
"nic": "ib0",
"bootproto": "dhcp",
"blockingdhcp": true,
"bridged": "false",
"mtu": "65520"
}
---------------------------------
------------- 4.0 ---------------
[root(a)node4.0]# cat /etc/sysconfig/network-scripts/ifcfg-ib0
# Generated by VDSM version 4.18.5.1-1.el7.centos
DEVICE=ib0
BRIDGE=ibstorage
ONBOOT=yes
MTU=65520
NM_CONTROLLED=no
IPV6INIT=no
[root(a)node4.0]# cat /var/lib/vdsm/persistence/netconf/nets/ibstorage
{
"ipv6autoconf": false,
"nic": "ib0",
"mtu": 65520,
"switch": "legacy",
"dhcpv6": false,
"stp": false,
"bridged": true,
"defaultRoute": false
}
[root(a)node4.0]# cat /var/lib/vdsm/persistence/netconf/nets/ib0
{
"ipv6autoconf": false,
"nic": "ib0",
"mtu": 65520,
"switch": "legacy",
"dhcpv6": false,
"stp": false,
"bridged": true,
"defaultRoute": false
}
[root(a)node4.0]# cat /etc/sysconfig/network-scripts/ifcfg-ibstorage
# Generated by VDSM version 4.18.5.1-1.el7.centos
DEVICE=ibstorage
TYPE=Bridge
DELAY=0
STP=off
ONBOOT=yes
DEFROUTE=no
NM_CONTROLLED=no
IPV6INIT=no
MTU=65520
---------------------------------
So it seems that 4.0 generates more files and wants to bridge an ethernet
unbridgeable link. Steps to attempt to configure the connection for 4.0
was the
same done successfully for 3.6.
Thanks again
On 07/04/2016 03:25 PM, Arman Khalatyan wrote:
> As I remember the IBoIP is not fully implemented Ethernet protocol, it
does not
> support bridging.
>
>
> ***********************************************************
>
> Dr. Arman Khalatyan eScience -SuperComputing
> Leibniz-Institut für Astrophysik Potsdam (AIP)
> An der Sternwarte 16, 14482 Potsdam, Germany
>
> ***********************************************************
>
> On Mon, Jul 4, 2016 at 2:58 PM, Yevgeny Zaspitsky <yzaspits(a)redhat.com
> <mailto:yzaspits@redhat.com>> wrote:
>
> Hello,
>
> What's "couple of vdsm hooks"? Could you provide the source code
of
those?
>
> Regards,
> Yevgeny
>
> On Mon, Jul 4, 2016 at 1:43 PM, Giorgio Biacchi <giorgio(a)di.unimi.it
> <mailto:giorgio@di.unimi.it>> wrote:
>
> Hi list,
> I'm trying a test installation of the new Ovirt Node Image and
engine before
> upgrading my 3.6 installation.
>
> In my current environment every hypervisor access the NFS shared
data
> domain via
> infiniband. With a couple of vdsm hooks I was able to set the
infiniband
> link in
> connected mode and set a MTU of 65520. In my case the infiniband
link is
> used as
> a common NIC, I don't use RDMA, only IP over infiniband to
access the
> data domain.
>
> Now, when I attempt to setup host networking, the system try to
bridge the
> connection "ibstorage" (defined in networks tab) with ib0
device. But a non
> ethernet device cannot be bridged that way and host networking
setup fails.
>
> Any idea or suggestion? With 3.6 (node and engine) all was fine.
>
> Thanks
> --
> gb
>
> PGP Key: http://pgp.mit.edu/
> Primary key fingerprint: C510 0765 943E EBED A4F2 69D3 16CC DC90
B9CB 0F34
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users
>
>
--
gb
PGP Key: http://pgp.mit.edu/
Primary key fingerprint: C510 0765 943E EBED A4F2 69D3 16CC DC90 B9CB 0F34
10:28 a.m.
Thanks,
setting the link as non VM networks let me configure the interface as needed.
Regards
On 07/04/2016 04:16 PM, Yevgeny Zaspitsky wrote:
Infiniband should not be bridged.
Ovirt tries to build a bridge over a NIC if the network is a VM one. Please
verify that it isn't.
On Mon, Jul 4, 2016 at 5:10 PM, Giorgio Biacchi <giorgio(a)di.unimi.it
<mailto:giorgio@di.unimi.it>> wrote:
As I said, Infiniband is not ethernet so it cannot be L2 bridged. This is normal
and I understand why it can't be done. I'd like to know if there's a way
to
configure host networking for infiniband link without bridging because is not
necessary to have a bridge there. In addition to this, 3.6 behavior was
different. Here's all the relevant configuration files generated by 3.6 and 4.0:
------------- 3.6 ---------------
[root(a)node3.6]# cat /etc/sysconfig/network-scripts/ifcfg-ib0
# Generated by VDSM version 4.17.11-7.gitc0752ac.el7
DEVICE=ib0
HWADDR=80:00:00:48:fe:80:00:00:00:00:00:00:00:02:c9:03:00:0a:06:9f
ONBOOT=yes
BOOTPROTO=dhcp
MTU=65520
DEFROUTE=no
NM_CONTROLLED=no
IPV6INIT=no
[root(a)node3.6]# cat /var/lib/vdsm/persistence/netconf/nets/ibstorage
{
"nic": "ib0",
"bootproto": "dhcp",
"blockingdhcp": true,
"bridged": "false",
"mtu": "65520"
}
---------------------------------
------------- 4.0 ---------------
[root(a)node4.0]# cat /etc/sysconfig/network-scripts/ifcfg-ib0
# Generated by VDSM version 4.18.5.1-1.el7.centos
DEVICE=ib0
BRIDGE=ibstorage
ONBOOT=yes
MTU=65520
NM_CONTROLLED=no
IPV6INIT=no
[root(a)node4.0]# cat /var/lib/vdsm/persistence/netconf/nets/ibstorage
{
"ipv6autoconf": false,
"nic": "ib0",
"mtu": 65520,
"switch": "legacy",
"dhcpv6": false,
"stp": false,
"bridged": true,
"defaultRoute": false
}
[root(a)node4.0]# cat /var/lib/vdsm/persistence/netconf/nets/ib0
{
"ipv6autoconf": false,
"nic": "ib0",
"mtu": 65520,
"switch": "legacy",
"dhcpv6": false,
"stp": false,
"bridged": true,
"defaultRoute": false
}
[root(a)node4.0]# cat /etc/sysconfig/network-scripts/ifcfg-ibstorage
# Generated by VDSM version 4.18.5.1-1.el7.centos
DEVICE=ibstorage
TYPE=Bridge
DELAY=0
STP=off
ONBOOT=yes
DEFROUTE=no
NM_CONTROLLED=no
IPV6INIT=no
MTU=65520
---------------------------------
So it seems that 4.0 generates more files and wants to bridge an ethernet
unbridgeable link. Steps to attempt to configure the connection for 4.0 was the
same done successfully for 3.6.
Thanks again
On 07/04/2016 03:25 PM, Arman Khalatyan wrote:
> As I remember the IBoIP is not fully implemented Ethernet protocol, it does not
> support bridging.
>
>
> ***********************************************************
>
> Dr. Arman Khalatyan eScience -SuperComputing
> Leibniz-Institut für Astrophysik Potsdam (AIP)
> An der Sternwarte 16, 14482 Potsdam, Germany
>
> ***********************************************************
>
> On Mon, Jul 4, 2016 at 2:58 PM, Yevgeny Zaspitsky <yzaspits(a)redhat.com
<mailto:yzaspits@redhat.com>
> <mailto:yzaspits@redhat.com <mailto:yzaspits@redhat.com>>>
wrote:
>
> Hello,
>
> What's "couple of vdsm hooks"? Could you provide the source
code of those?
>
> Regards,
> Yevgeny
>
> On Mon, Jul 4, 2016 at 1:43 PM, Giorgio Biacchi <giorgio(a)di.unimi.it
<mailto:giorgio@di.unimi.it>
> <mailto:giorgio@di.unimi.it <mailto:giorgio@di.unimi.it>>>
wrote:
>
> Hi list,
> I'm trying a test installation of the new Ovirt Node Image and
engine before
> upgrading my 3.6 installation.
>
> In my current environment every hypervisor access the NFS shared data
> domain via
> infiniband. With a couple of vdsm hooks I was able to set the
infiniband
> link in
> connected mode and set a MTU of 65520. In my case the infiniband link
is
> used as
> a common NIC, I don't use RDMA, only IP over infiniband to access
the
> data domain.
>
> Now, when I attempt to setup host networking, the system try to bridge
the
> connection "ibstorage" (defined in networks tab) with ib0
device. But a non
> ethernet device cannot be bridged that way and host networking setup
fails.
>
> Any idea or suggestion? With 3.6 (node and engine) all was fine.
>
> Thanks
> --
> gb
>
> PGP Key: http://pgp.mit.edu/
> Primary key fingerprint: C510 0765 943E EBED A4F2 69D3 16CC DC90 B9CB
0F34
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
<mailto:Users@ovirt.org
<mailto:Users@ovirt.org>>
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org> <mailto:Users@ovirt.org
<mailto:Users@ovirt.org>>
> http://lists.ovirt.org/mailman/listinfo/users
>
>
--
gb
PGP Key: http://pgp.mit.edu/
Primary key fingerprint: C510 0765 943E EBED A4F2 69D3 16CC DC90 B9CB 0F34
--
gb
PGP Key: http://pgp.mit.edu/
Primary key fingerprint: C510 0765 943E EBED A4F2 69D3 16CC DC90 B9CB 0F34
3098
days inactive
3098
days old
6 comments
3 participants
participants (3)
-
Arman Khalatyan -
Giorgio Biacchi -
Yevgeny Zaspitsky