[ovirt-users] Re: Creating a static route

Tuesday, 12 February 2019

Will do.  Thanks again for all your help!!  Looking forward to a confirmation of the fix
or a different solution.  Either way, I know I have a working fix.

[Count On EMC]<http://www.emcins.com/>

Doug Maxfield | Senior Operating Systems Analyst
EMC Insurance Companies
717 Mulberry St | Des Moines, IA 50265
Tel: 515.345.4507 | Fax: 866.331.1522
Doug.L.Maxfield@EMCIns.com<mailto:Doug.L.Maxfield@EMCIns.com> |
www.emcins.com<http://www.emcins.com/>

From: Lev Veyde <lveyde(a)redhat.com&gt;
Sent: Tuesday, February 12, 2019 3:09 PM
To: Doug Maxfield <Doug.L.Maxfield(a)EMCIns.com&gt;
Cc: users <users(a)ovirt.org&gt;
Subject: Re: [ovirt-users] Creating a static route

Hi Doug,
Just a small note - the commands in the rc.local should be placed *before* the very last
line (the touch ... command).
Thanks in advance,

On Tue, Feb 12, 2019 at 10:56 PM Lev Veyde
<lveyde@redhat.com<mailto:lveyde@redhat.com>> wrote:
Hi Doug,
Thanks, you're welcome!
It's generally possible to add persistent rules by either using the legacy
/etc/rc.d/rc.local boot script (chmod +x /etc/rc.d/rc.local and put the relevant commands
at the end of the file, that way you can actually run any commands) or by using the
/etc/sysconfig/network-scripts/rule-<interface name> e.g. rule-ovirtmgmt or
rule-eno3 (just put in the file "to 172.22.20.31 priority 32764" - without
quotes), however of course this needs to be tested to verify that it works.
I forwarded the issue to the network team and hopefully they will update soon.

Thanks in advance,

On Tue, Feb 12, 2019 at 10:17 PM Doug Maxfield
<Doug.L.Maxfield@emcins.com<mailto:Doug.L.Maxfield@emcins.com>> wrote:
Lev,
You have no idea how helpful you have been!!!!!!  I’ve been having this issue for over 2
months and starting to cause major problems.

Please let me know if you find a better solution.  I’m going to wait to hear back from you
before implementing.

If we go with this workaround, is there a way to add the ip rule so it’s persistent?  I
know how to make the ip route add persistent.

[Count On EMC]<http://www.emcins.com/>

Doug Maxfield | Senior Operating Systems Analyst
EMC Insurance Companies
717 Mulberry St | Des Moines, IA 50265
Tel: 515.345.4507 | Fax: 866.331.1522
Doug.L.Maxfield@EMCIns.com<mailto:Doug.L.Maxfield@EMCIns.com> |
www.emcins.com<http://www.emcins.com/>

From: Lev Veyde <lveyde@redhat.com<mailto:lveyde@redhat.com>>
Sent: Tuesday, February 12, 2019 2:14 PM
To: Doug Maxfield
<Doug.L.Maxfield@EMCIns.com<mailto:Doug.L.Maxfield@EMCIns.com>>
Cc: users <users@ovirt.org<mailto:users@ovirt.org>>
Subject: Re: [ovirt-users] Creating a static route

Hi Doug,
Yes, it looks like it did.
I'll try to catch somebody from my local oVirt/RHV networking team to take a look at
this issue, maybe there is a better solution for that issue than my workaround.

But meanwhile you can use this workaround (ip route add + ip rule add), just please be
aware that these should be run after each host reboot, since these commands are not
persistent.

Thanks in advance,

On Tue, Feb 12, 2019 at 10:07 PM Doug Maxfield
<Doug.L.Maxfield@emcins.com<mailto:Doug.L.Maxfield@emcins.com>> wrote:
Lev,
Here’s the results.  It appears to fix the issue!!

From non-working system:

-bash-4.2# ip rule add to 172.22.20.31 priority 32764

-bash-4.2# traceroute 172.22.20.31
traceroute to 172.22.20.31 (172.22.20.31), 30 hops max, 60 byte packets
1  CenteraAmesAN1.emcins.com<http://CenteraAmesAN1.emcins.com> (172.21.12.61)  0.487
ms  1.082 ms  1.313 ms
2  192.168.90.9 (192.168.90.9)  1.507 ms  2.090 ms  2.294 ms
3  pdputopcomm01.emcins.com<http://pdputopcomm01.emcins.com> (172.22.20.31)  1.403
ms  1.401 ms  1.348 ms

[Count On EMC]<http://www.emcins.com/>

Doug Maxfield | Senior Operating Systems Analyst
EMC Insurance Companies
717 Mulberry St | Des Moines, IA 50265
Tel: 515.345.4507 | Fax: 866.331.1522
Doug.L.Maxfield@EMCIns.com<mailto:Doug.L.Maxfield@EMCIns.com> |
www.emcins.com<http://www.emcins.com/>

From: Lev Veyde <lveyde@redhat.com<mailto:lveyde@redhat.com>>
Sent: Tuesday, February 12, 2019 2:01 PM
To: Doug Maxfield
<Doug.L.Maxfield@EMCIns.com<mailto:Doug.L.Maxfield@EMCIns.com>>
Cc: users <users@ovirt.org<mailto:users@ovirt.org>>
Subject: Re: [ovirt-users] Creating a static route

Hi Doug,
OK, I think that I have a guess of what may have happened here...
Can you please run the following command on the (non-working) host:

ip rule add to 172.22.20.31 priority 32764

This needs to be run in addition to the ip route add ... command.

And then please run the traceroute command and let me know if that fixed the issue.

Thanks in advance,

On Tue, Feb 12, 2019 at 9:30 PM Doug Maxfield
<Doug.L.Maxfield@emcins.com<mailto:Doug.L.Maxfield@emcins.com>> wrote:
Lev,
Here you go.

From non-working system

-bash-4.2# ip rule
0:      from all lookup local
32764:  from all to 172.21.0.0/16<http://172.21.0.0/16> iif ovirtmgmt lookup
2887058719
32765:  from 172.21.0.0/16<http://172.21.0.0/16> lookup 2887058719
32766:  from all lookup main
32767:  from all lookup default

From working system

[root@paputopcomm04 ~]# ip rule
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

[Count On EMC]<http://www.emcins.com/>

Doug Maxfield | Senior Operating Systems Analyst
EMC Insurance Companies
717 Mulberry St | Des Moines, IA 50265
Tel: 515.345.4507 | Fax: 866.331.1522
Doug.L.Maxfield@EMCIns.com<mailto:Doug.L.Maxfield@EMCIns.com> |
www.emcins.com<http://www.emcins.com/>

From: Lev Veyde <lveyde@redhat.com<mailto:lveyde@redhat.com>>
Sent: Tuesday, February 12, 2019 12:35 PM
To: Doug Maxfield
<Doug.L.Maxfield@EMCIns.com<mailto:Doug.L.Maxfield@EMCIns.com>>
Cc: users <users@ovirt.org<mailto:users@ovirt.org>>
Subject: Re: [ovirt-users] Creating a static route

Hi Doug,
That shouldn't really matter, since ovirtmgmt is just a bridged interface, and unless
I'm missing something it shouldn't really affect routing.
Can you please also provide the output of "ip rule" from both working and
non-working hosts ?

Thanks in advance,

On Tue, Feb 12, 2019 at 8:01 PM Doug Maxfield
<Doug.L.Maxfield@emcins.com<mailto:Doug.L.Maxfield@emcins.com>> wrote:
Lev,
Here’s the ip route from the server that worked and didn’t work

IP Route from server that worked

[root@paputopcomm04 ~]# ip route
default via 172.21.0.250 dev eno3
169.254.0.0/16<http://169.254.0.0/16> dev eno3 scope link metric 1002
169.254.0.0/16<http://169.254.0.0/16> dev eno4 scope link metric 1005
172.17.0.0/16<http://172.17.0.0/16> dev eno4 proto kernel scope link src
172.17.53.18
172.21.0.0/16<http://172.21.0.0/16> dev eno3 proto kernel scope link src
172.21.5.34
172.22.20.31 via 172.21.12.61 dev eno3
172.26.0.0/16<http://172.26.0.0/16> dev eno1 proto kernel scope link src
172.26.5.34

IP Route from server that didn't work

-bash-4.2# ip route
default via 172.21.0.250 dev ovirtmgmt
169.254.0.0/16<http://169.254.0.0/16> dev eno4 scope link metric 1004
169.254.0.0/16<http://169.254.0.0/16> dev ovirtmgmt scope link metric 1021
172.17.0.0/16<http://172.17.0.0/16> dev eno4 proto kernel scope link src
172.17.53.15
172.21.0.0/16<http://172.21.0.0/16> dev ovirtmgmt proto kernel scope link src
172.21.5.31
172.22.20.31 via 172.21.12.61 dev ovirtmgmt
172.26.0.0/16<http://172.26.0.0/16> dev eno1 proto kernel scope link src
172.26.5.31

We have had our network team look and they don’t know.  As I said, the was setup by
Commvault.  It’s almost like the ovirtmgmt interface will not allow a route add due to the
way it’s configured?   They, Commvault, are trying different things within their software
to get this working, but it’s causing us more problems.  I figured I would take a shot and
see if someone, who works with the oVirt software, would have any ideas.

[Count On EMC]<http://www.emcins.com/>

Doug Maxfield | Senior Operating Systems Analyst
EMC Insurance Companies
717 Mulberry St | Des Moines, IA 50265
Tel: 515.345.4507 | Fax: 866.331.1522
Doug.L.Maxfield@EMCIns.com<mailto:Doug.L.Maxfield@EMCIns.com> |
www.emcins.com<http://www.emcins.com/>

From: Lev Veyde <lveyde@redhat.com<mailto:lveyde@redhat.com>>
Sent: Tuesday, February 12, 2019 11:51 AM
To: Doug Maxfield
<Doug.L.Maxfield@EMCIns.com<mailto:Doug.L.Maxfield@EMCIns.com>>
Cc: users <users@ovirt.org<mailto:users@ovirt.org>>
Subject: Re: [ovirt-users] Creating a static route

Hi Doug,
This is indeed quite weird...
It looks like the OS for some reason disregards the static route that you have added.
BTW, have you verified that it's indeed was added by running "ip route"
following the "ip route add ..." command ?
Maybe somebody from the network team has an idea of what may be the cause of this
issue...

Thanks in advance,

On Tue, Feb 12, 2019 at 7:36 PM Doug Maxfield
<Doug.L.Maxfield@emcins.com<mailto:Doug.L.Maxfield@emcins.com>> wrote:
Lev,
I’m showing you the same outputs (ip a, ip route, traceroute, ip route add, and
traceroute) from another server in this group that doesn’t use the ovirtmgmt for it’s
default interface.

[root@paputopcomm04 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8<http://127.0.0.1/8> scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 90:1b:0e:db:bc:c0 brd ff:ff:ff:ff:ff:ff
    inet 172.21.5.34/16<http://172.21.5.34/16> brd 172.21.255.255 scope global
eno3
       valid_lft forever preferred_lft forever
    inet6 fe80::921b:eff:fedb:bcc0/64 scope link
       valid_lft forever preferred_lft forever
3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 90:1b:0e:e8:da:03 brd ff:ff:ff:ff:ff:ff
    inet 172.26.5.34/16<http://172.26.5.34/16> brd 172.26.255.255 scope global
eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::921b:eff:fee8:da03/64 scope link
       valid_lft forever preferred_lft forever
4: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen
1000
    link/ether 90:1b:0e:e8:da:04 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 90:1b:0e:db:bc:c1 brd ff:ff:ff:ff:ff:ff
    inet 172.17.53.18/16<http://172.17.53.18/16> brd 172.17.255.255 scope global
eno4
       valid_lft forever preferred_lft forever
    inet6 fe80::921b:eff:fedb:bcc1/64 scope link
       valid_lft forever preferred_lft forever
[root@paputopcomm04 ~]#
[root@paputopcomm04 ~]#
[root@paputopcomm04 ~]# ip route
default via 172.21.0.250 dev eno3
169.254.0.0/16<http://169.254.0.0/16> dev eno3 scope link metric 1002
169.254.0.0/16<http://169.254.0.0/16> dev eno4 scope link metric 1005
172.17.0.0/16<http://172.17.0.0/16> dev eno4 proto kernel scope link src
172.17.53.18
172.21.0.0/16<http://172.21.0.0/16> dev eno3 proto kernel scope link src
172.21.5.34
172.26.0.0/16<http://172.26.0.0/16> dev eno1 proto kernel scope link src
172.26.5.34
[root@paputopcomm04 ~]#
[root@paputopcomm04 ~]#
[root@paputopcomm04 ~]# traceroute 172.22.20.31
traceroute to 172.22.20.31 (172.22.20.31), 30 hops max, 60 byte packets
1  ames-acc1-agg-1.emcins.com<http://ames-acc1-agg-1.emcins.com> (172.21.0.246) 
1.081 ms  1.816 ms  1.832 ms
2  192.168.200.30 (192.168.200.30)  0.639 ms 192.168.200.14 (192.168.200.14)  1.063 ms 
1.577 ms
3  192.168.200.101 (192.168.200.101)  3.110 ms
ames-sw-wanrtr.emcins.com<http://ames-sw-wanrtr.emcins.com> (192.168.200.97)  3.191
ms  3.277 ms
4  192.168.100.26 (192.168.100.26)  4.146 ms  4.273 ms  4.394 ms
5  192.168.100.98 (192.168.100.98)  1.893 ms  2.364 ms  2.397 ms
6  192.168.100.50 (192.168.100.50)  5.707 ms 192.168.100.33 (192.168.100.33)  5.134 ms
192.168.100.50 (192.168.100.50)  4.105 ms
7  pdputopcomm01.emcins.com<http://pdputopcomm01.emcins.com> (172.22.20.31)  1.359
ms  1.374 ms  1.351 ms
[root@paputopcomm04 ~]#
[root@paputopcomm04 ~]#
[root@paputopcomm04 ~]#
[root@paputopcomm04 ~]#
[root@paputopcomm04 ~]# ip route add 172.22.20.31 via 172.21.12.61
[root@paputopcomm04 ~]#
[root@paputopcomm04 ~]#
[root@paputopcomm04 ~]#
[root@paputopcomm04 ~]# traceroute 172.22.20.31
traceroute to 172.22.20.31 (172.22.20.31), 30 hops max, 60 byte packets
1  CenteraAmesAN1.emcins.com<http://CenteraAmesAN1.emcins.com> (172.21.12.61)  6.184
ms  6.475 ms  6.686 ms
2  192.168.90.9 (192.168.90.9)  1.770 ms  1.979 ms  2.189 ms
3  pdputopcomm01.emcins.com<http://pdputopcomm01.emcins.com> (172.22.20.31)  1.405
ms *  1.354 ms

[Count On EMC]<http://www.emcins.com/>

Doug Maxfield | Senior Operating Systems Analyst
EMC Insurance Companies
717 Mulberry St | Des Moines, IA 50265
Tel: 515.345.4507 | Fax: 866.331.1522
Doug.L.Maxfield@EMCIns.com<mailto:Doug.L.Maxfield@EMCIns.com> |
www.emcins.com<http://www.emcins.com/>

From: Lev Veyde <lveyde@redhat.com<mailto:lveyde@redhat.com>>
Sent: Tuesday, February 12, 2019 11:07 AM
To: Doug Maxfield
<Doug.L.Maxfield@EMCIns.com<mailto:Doug.L.Maxfield@EMCIns.com>>
Cc: users <users@ovirt.org<mailto:users@ovirt.org>>
Subject: Re: [ovirt-users] Creating a static route

Hi Doug,
In most cases it should not be required, especially as my guess (again it would be easier
if you could post the output of the "ip a" and "ip route") is that the
eno1 is part of the ovirtmgmt bridge.
What is the output of the "traceroute 172.22.20.31" that you get on the host
after you have added the static route?
Thanks in advance,

On Tue, Feb 12, 2019 at 6:58 PM Doug Maxfield
<Doug.L.Maxfield@emcins.com<mailto:Doug.L.Maxfield@emcins.com>> wrote:
Lev,
One other question, since there are multiple enabled interfaces (ovirtmgmt and eno1), do I
need to specify an interface for the route add.

[Count On EMC]<http://www.emcins.com/>

Doug Maxfield | Senior Operating Systems Analyst
EMC Insurance Companies
717 Mulberry St | Des Moines, IA 50265
Tel: 515.345.4507 | Fax: 866.331.1522
Doug.L.Maxfield@EMCIns.com<mailto:Doug.L.Maxfield@EMCIns.com> |
www.emcins.com<http://www.emcins.com/>

From: Lev Veyde <lveyde@redhat.com<mailto:lveyde@redhat.com>>
Sent: Tuesday, February 12, 2019 10:50 AM
To: Doug Maxfield
<Doug.L.Maxfield@EMCIns.com<mailto:Doug.L.Maxfield@EMCIns.com>>
Cc: users <users@ovirt.org<mailto:users@ovirt.org>>
Subject: Re: [ovirt-users] Creating a static route

Hi Doug,
What do you mean by "static route is refused"?
What is the error message(s) you see?

Can you please try to add the route with the following command:
ip route add 172.22.20.31 via 172.21.12.61

If that doesn't work, then please provide more details, i.e. what is the exact error
message(s) you see, the version of your OS and oVirt, output of "ip a" and
"ip route" on your host(s), so that it will be easier to help you.

Thanks in advance,

On Tue, Feb 12, 2019 at 6:43 PM Doug Maxfield
<Doug.L.Maxfield@emcins.com<mailto:Doug.L.Maxfield@emcins.com>> wrote:
Lev,
Thanks for your response.

If I attempt to manually add the route using route add:

route add -net 172.22.20.31 netmask 255.255.255.255 gw 172.21.12.61 dev ovirtmgmt

This static route is refused and all traffic is routed over the default gateway.

But if I use a server that doesn’t use the ovirtmgmt interface and set the same static
route, the required data is routed correctly over that static route.

[Count On EMC]<http://www.emcins.com/>

Doug Maxfield | Senior Operating Systems Analyst
EMC Insurance Companies
717 Mulberry St | Des Moines, IA 50265
Tel: 515.345.4507 | Fax: 866.331.1522
Doug.L.Maxfield@EMCIns.com<mailto:Doug.L.Maxfield@EMCIns.com> |
www.emcins.com<http://www.emcins.com/>

From: Lev Veyde <lveyde@redhat.com<mailto:lveyde@redhat.com>>
Sent: Tuesday, February 12, 2019 10:34 AM
To: Doug Maxfield
<Doug.L.Maxfield@EMCIns.com<mailto:Doug.L.Maxfield@EMCIns.com>>
Cc: users <users@ovirt.org<mailto:users@ovirt.org>>
Subject: Re: [ovirt-users] Creating a static route

Hi Doug,
What is the exact problem you are having while attempting to add a static route on the
hosts?

Thanks in advance,

On Tue, Feb 12, 2019 at 5:23 PM
<doug.l.maxfield@emcins.com<mailto:doug.l.maxfield@emcins.com>> wrote:
Good Morning,
New to oVirt.  We are using this with a backup solution from Commvault.  The issue that we
are having is that we need to setup a static route for specific data between 2 remote
sites.  The ovirtmgmt is configured with the correct IP and gateway for the server.  We
need to route data over a different gateway so that we don't "max out" the
default network connection between the 2 sites.  Example below

Ovirtmgmt IP - 172.21.5.31
Gateway IP - 172.21.0.250

We need any traffic with a destination of 172.22.20.31(Remote site) to route over this
gateway, 172.21.12.61

There are multiple servers in this configuration.  Servers that are not using the
ovirtmgmt interface for their default, we are able to route the data.  The only problem is
with attempting to setup a different static route on the ovirtmgmt interfaces.

Thanks in advance for any help!!!

Doug
_______________________________________________
Users mailing list -- users@ovirt.org<mailto:users@ovirt.org>
To unsubscribe send an email to
users-leave@ovirt.org<mailto:users-leave@ovirt.org>
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/WGXSLHAP3X5...

--

Lev Veyde

Software Engineer, RHCE | RHCVA | MCITP

Red Hat Israel

<https://www.redhat.com>