[ovirt-users] 3.6 looses network on reboot

[Dan Kenigsberg]

This sounds very bad. Changing the subject, so the wider, more
problematic issue is visible.

Did any other user see this behavior?

On Mon, Feb 29, 2016 at 06:27:46AM +0000, David LeVene wrote:
> Hi Dan,
> 
> Answers as follows;
> 
> # rpm -qa | grep -i vdsm
> vdsm-jsonrpc-4.17.18-1.el7.noarch
> vdsm-hook-vmfex-4.17.18-1.el7.noarch
> vdsm-infra-4.17.18-1.el7.noarch
> vdsm-4.17.18-1.el7.noarch
> vdsm-python-4.17.18-1.el7.noarch
> vdsm-yajsonrpc-4.17.18-1.el7.noarch
> vdsm-cli-4.17.18-1.el7.noarch
> vdsm-xmlrpc-4.17.18-1.el7.noarch
> vdsm-hook-vmfex-dev-4.17.18-1.el7.noarch
> 
> 
> There was in this folder ifcfg-ovirtmgnt bridge setup, and also route-ovirtmgnt & rule-ovirtmgmt.. but they were removed after the reboot.
> 
> # ls -althr | grep ifcfg
> -rw-r--r--. 1 root root  254 Sep 16 21:21 ifcfg-lo
> -rw-r--r--. 1 root root  120 Feb 25 14:07 ifcfg-enp7s0f0
> -rw-rw-r--. 1 root root  174 Feb 25 14:40 ifcfg-enp6s0
> 
> I think I modified ifcfg-enp6s0 to get networking up again (eg was set to bridge.. but the bridge wasn't configured).. it was a few days ago.. if it's important I can reboot the box again to see what state it comes up with.
> 
> # cat ifcfg-enp6s0
> BOOTPROTO="none"
> IPADDR="10.80.10.117"
> NETMASK="255.255.255.0"
> GATEWAY="10.80.10.1"
> DEVICE="enp6s0"
> HWADDR="00:25:b5:00:0b:4f"
> ONBOOT=yes
> PEERDNS=yes
> PEERROUTES=yes
> MTU=1500
> 
> # cat ifcfg-enp7s0f0
> # Generated by VDSM version 4.17.18-1.el7
> DEVICE=enp7s0f0
> ONBOOT=yes
> MTU=1500
> HWADDR=00:25:b5:00:0b:0f
> NM_CONTROLLED=no
> 
> # find /var/lib/vdsm/persistence
> /var/lib/vdsm/persistence
> /var/lib/vdsm/persistence/netconf
> /var/lib/vdsm/persistence/netconf.1456371473833165545
> /var/lib/vdsm/persistence/netconf.1456371473833165545/nets
> /var/lib/vdsm/persistence/netconf.1456371473833165545/nets/ovirtmgmt
> 
> # cat /var/lib/vdsm/persistence/netconf.1456371473833165545/nets/ovirtmgmt
> {
>     "nic": "enp6s0",
>     "ipaddr": "10.80.10.117",
>     "mtu": "1500",
>     "netmask": "255.255.255.0",
>     "STP": "no",
>     "bridged": "true",
>     "gateway": "10.80.10.1",
>     "defaultRoute": true
> }
> 
> Supervdsm log is attached.

Have you editted ifcfg-ovirtmgmt manually? Can you somehow reproduce it,
and share its content?
Do you have NetworkManager running? which version?

It seems that Vdsm has two bugs: on boot, initscripts end up setting an
ipv6 address that Vdsm never requested.

    restore-net::INFO::2016-02-25 14:14:58,024::vdsm-restore-net-config::261::root::(_find_changed_or_missing) ovirtmgmt is different or missing from persistent configuration. current: {'nic': 'enp6s0', 'dhcpv6': False, 'ipaddr': '10.80.10.117', 'mtu': '1500', 'netmask': '255.255.255.0', 'bootproto': 'none', 'stp': False, 'bridged': True, 'ipv6addr': ['2400:7d00:110:3:225:b5ff:fe00:b4f/64'], 'gateway': '10.80.10.1', 'defaultRoute': True}, persisted: {u'nic': u'enp6s0', 'dhcpv6': False, u'ipaddr': u'10.80.10.117', u'mtu': '1500', u'netmask': u'255.255.255.0', 'bootproto': 'none', 'stp': False, u'bridged': True, u'gateway': u'10.80.10.1', u'defaultRoute': True}


Then, Vdsm tries to drop the
unsolicited address, but fails. Both must be fixed ASAP.

    restore-net::ERROR::2016-02-25 14:14:59,490::__init__::58::root::(__exit__) Failed rollback transaction last known good network.
    Traceback (most recent call last):
      File "/usr/share/vdsm/network/api.py", line 918, in setupNetworks
        keep_bridge=keep_bridge)
      File "/usr/share/vdsm/network/api.py", line 222, in wrapped
        ret = func(**attrs)
      File "/usr/share/vdsm/network/api.py", line 502, in _delNetwork
        configurator.removeQoS(net_ent)
      File "/usr/share/vdsm/network/configurators/__init__.py", line 122, in removeQoS
        qos.remove_outbound(top_device)
      File "/usr/share/vdsm/network/configurators/qos.py", line 60, in remove_outbound
        device, pref=_NON_VLANNED_ID if vlan_tag is None else vlan_tag)
      File "/usr/share/vdsm/network/tc/filter.py", line 31, in delete
        _wrapper.process_request(command)
      File "/usr/share/vdsm/network/tc/_wrapper.py", line 38, in process_request
        raise TrafficControlException(retcode, err, command)
    TrafficControlException: (None, 'Message truncated', ['/usr/sbin/tc', 'filter', 'del', 'dev', 'enp6s0', 'pref', '5000'])

Regards,
Dan.