Re: [ovirt-users] 3.6 looses network on reboot

Hi Dan, I missed the email as the subject line changed! So we use and run IPv6 in our network - not sure if this is related. The Addresses are handed out via SLAAC so that would be where the IPv6 address is coming from. My memory is a bit sketchy... but I think if I remove the vmfex/SRIOV vNIC and only run with the one vNIC it works fine, it's when I bring the second NIC into play with SRIOV the issues arise. Answers inline. -----Original Message----- From: Dan Kenigsberg [mailto:danken@redhat.com] Sent: Tuesday, March 01, 2016 00:28 To: David LeVene <David.LeVene(a)blackboard.com&gt; Cc: edwardh(a)redhat.com; users(a)ovirt.org Subject: Re: [ovirt-users] 3.6 looses network on reboot This sounds very bad. Changing the subject, so the wider, more problematic issue is visible. Did any other user see this behavior? On Mon, Feb 29, 2016 at 06:27:46AM +0000, David LeVene wrote: > Hi Dan, > > Answers as follows; > > # rpm -qa | grep -i vdsm > vdsm-jsonrpc-4.17.18-1.el7.noarch > vdsm-hook-vmfex-4.17.18-1.el7.noarch > vdsm-infra-4.17.18-1.el7.noarch > vdsm-4.17.18-1.el7.noarch > vdsm-python-4.17.18-1.el7.noarch > vdsm-yajsonrpc-4.17.18-1.el7.noarch > vdsm-cli-4.17.18-1.el7.noarch > vdsm-xmlrpc-4.17.18-1.el7.noarch > vdsm-hook-vmfex-dev-4.17.18-1.el7.noarch > > > There was in this folder ifcfg-ovirtmgnt bridge setup, and also route-ovirtmgnt & rule-ovirtmgmt.. but they were removed after the reboot. > > # ls -althr | grep ifcfg > -rw-r--r--. 1 root root 254 Sep 16 21:21 ifcfg-lo -rw-r--r--. 1 root > root 120 Feb 25 14:07 ifcfg-enp7s0f0 -rw-rw-r--. 1 root root 174 Feb > 25 14:40 ifcfg-enp6s0 > > I think I modified ifcfg-enp6s0 to get networking up again (eg was set to bridge.. but the bridge wasn't configured).. it was a few days ago.. if it's important I can reboot the box again to see what state it comes up with. > > # cat ifcfg-enp6s0 > BOOTPROTO="none" > IPADDR="10.80.10.117" > NETMASK="255.255.255.0" > GATEWAY="10.80.10.1" > DEVICE="enp6s0" > HWADDR="00:25:b5:00:0b:4f" > ONBOOT=yes > PEERDNS=yes > PEERROUTES=yes > MTU=1500 > > # cat ifcfg-enp7s0f0 > # Generated by VDSM version 4.17.18-1.el7 > DEVICE=enp7s0f0 > ONBOOT=yes > MTU=1500 > HWADDR=00:25:b5:00:0b:0f > NM_CONTROLLED=no > > # find /var/lib/vdsm/persistence > /var/lib/vdsm/persistence > /var/lib/vdsm/persistence/netconf > /var/lib/vdsm/persistence/netconf.1456371473833165545 > /var/lib/vdsm/persistence/netconf.1456371473833165545/nets > /var/lib/vdsm/persistence/netconf.1456371473833165545/nets/ovirtmgmt > > # cat > /var/lib/vdsm/persistence/netconf.1456371473833165545/nets/ovirtmgmt > { > "nic": "enp6s0", > "ipaddr": "10.80.10.117", > "mtu": "1500", > "netmask": "255.255.255.0", > "STP": "no", > "bridged": "true", > "gateway": "10.80.10.1", > "defaultRoute": true > } > > Supervdsm log is attached. Have you editted ifcfg-ovirtmgmt manually? Nope Can you somehow reproduce it, and share its content? Yea, I should be able to reproduce it - just gotta fix it first (create the networking manually and get VDSM on-line). Also it’s a side project/investigation at the moment so time isn't on my side... Would it help if I take an sosreport before and after? I don’t' mine emailing these directly to yourself. Do you have NetworkManager running? which version? NM is disabled, but the version is... # rpm -q NetworkManager NetworkManager-1.0.6-27.el7.x86_64 # systemctl status NetworkManager.service ● NetworkManager.service - Network Manager Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; disabled; vendor preset: enabled) Active: inactive (dead) It seems that Vdsm has two bugs: on boot, initscripts end up setting an ipv6 address that Vdsm never requested As mentioned above this would have come from SLAAC which we have setup in our network restore-net::INFO::2016-02-25 14:14:58,024::vdsm-restore-net-config::261::root::(_find_changed_or_missing) ovirtmgmt is different or missing from persistent configuration. current: {'nic': 'enp6s0', 'dhcpv6': False, 'ipaddr': '10.80.10.117', 'mtu': '1500', 'netmask': '255.255.255.0', 'bootproto': 'none', 'stp': False, 'bridged': True, 'ipv6addr': ['2400:7d00:110:3:225:b5ff:fe00:b4f/64'], 'gateway': '10.80.10.1', 'defaultRoute': True}, persisted: {u'nic': u'enp6s0', 'dhcpv6': False, u'ipaddr': u'10.80.10.117', u'mtu': '1500', u'netmask': u'255.255.255.0', 'bootproto': 'none', 'stp': False, u'bridged': True, u'gateway': u'10.80.10.1', u'defaultRoute': True} Then, Vdsm tries to drop the unsolicited address, but fails. Both must be fixed ASAP. restore-net::ERROR::2016-02-25 14:14:59,490::__init__::58::root::(__exit__) Failed rollback transaction last known good network. Traceback (most recent call last): File "/usr/share/vdsm/network/api.py", line 918, in setupNetworks keep_bridge=keep_bridge) File "/usr/share/vdsm/network/api.py", line 222, in wrapped ret = func(**attrs) File "/usr/share/vdsm/network/api.py", line 502, in _delNetwork configurator.removeQoS(net_ent) File "/usr/share/vdsm/network/configurators/__init__.py", line 122, in removeQoS qos.remove_outbound(top_device) File "/usr/share/vdsm/network/configurators/qos.py", line 60, in remove_outbound device, pref=_NON_VLANNED_ID if vlan_tag is None else vlan_tag) File "/usr/share/vdsm/network/tc/filter.py", line 31, in delete _wrapper.process_request(command) File "/usr/share/vdsm/network/tc/_wrapper.py", line 38, in process_request raise TrafficControlException(retcode, err, command) TrafficControlException: (None, 'Message truncated', ['/usr/sbin/tc', 'filter', 'del', 'dev', 'enp6s0', 'pref', '5000']) Regards, Dan.