<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div>I did see it few times. The first reboot after new node setup sometimes fails to bring network up. It tries to remove network interface when it doesn't exist.</div>
<div><br>
</div>
<div>Steps to recover:</div>
<div>1. Remove /var/lib/vdsm/persistence/netconf</div>
<div>2. Remove /var/run/vdsm/netconf</div>
<div>3. Configure network manually</div>
<div>4. Start vdsmd service</div>
<div>5. Configure network again using web ui. Make sure that config is synced.</div>
<div><br>
</div>
<div>On Mon, 2016-02-29 at 15:58 +0200, Dan Kenigsberg wrote:</div>
<blockquote type="cite">
<pre>This sounds very bad. Changing the subject, so the wider, more
problematic issue is visible.
Did any other user see this behavior?
On Mon, Feb 29, 2016 at 06:27:46AM +0000, David LeVene wrote:
<blockquote type="cite">
Hi Dan,
Answers as follows;
# rpm -qa | grep -i vdsm
vdsm-jsonrpc-4.17.18-1.el7.noarch
vdsm-hook-vmfex-4.17.18-1.el7.noarch
vdsm-infra-4.17.18-1.el7.noarch
vdsm-4.17.18-1.el7.noarch
vdsm-python-4.17.18-1.el7.noarch
vdsm-yajsonrpc-4.17.18-1.el7.noarch
vdsm-cli-4.17.18-1.el7.noarch
vdsm-xmlrpc-4.17.18-1.el7.noarch
vdsm-hook-vmfex-dev-4.17.18-1.el7.noarch
There was in this folder ifcfg-ovirtmgnt bridge setup, and also route-ovirtmgnt & rule-ovirtmgmt.. but they were removed after the reboot.
# ls -althr | grep ifcfg
-rw-r--r--. 1 root root 254 Sep 16 21:21 ifcfg-lo
-rw-r--r--. 1 root root 120 Feb 25 14:07 ifcfg-enp7s0f0
-rw-rw-r--. 1 root root 174 Feb 25 14:40 ifcfg-enp6s0
I think I modified ifcfg-enp6s0 to get networking up again (eg was set to bridge.. but the bridge wasn't configured).. it was a few days ago.. if it's important I can reboot the box again to see what state it comes up with.
# cat ifcfg-enp6s0
BOOTPROTO="none"
IPADDR="10.80.10.117"
NETMASK="255.255.255.0"
GATEWAY="10.80.10.1"
DEVICE="enp6s0"
HWADDR="00:25:b5:00:0b:4f"
ONBOOT=yes
PEERDNS=yes
PEERROUTES=yes
MTU=1500
# cat ifcfg-enp7s0f0
# Generated by VDSM version 4.17.18-1.el7
DEVICE=enp7s0f0
ONBOOT=yes
MTU=1500
HWADDR=00:25:b5:00:0b:0f
NM_CONTROLLED=no
# find /var/lib/vdsm/persistence
/var/lib/vdsm/persistence
/var/lib/vdsm/persistence/netconf
/var/lib/vdsm/persistence/netconf.1456371473833165545
/var/lib/vdsm/persistence/netconf.1456371473833165545/nets
/var/lib/vdsm/persistence/netconf.1456371473833165545/nets/ovirtmgmt
# cat /var/lib/vdsm/persistence/netconf.1456371473833165545/nets/ovirtmgmt
{
"nic": "enp6s0",
"ipaddr": "10.80.10.117",
"mtu": "1500",
"netmask": "255.255.255.0",
"STP": "no",
"bridged": "true",
"gateway": "10.80.10.1",
"defaultRoute": true
}
Supervdsm log is attached.
</blockquote>
Have you editted ifcfg-ovirtmgmt manually? Can you somehow reproduce it,
and share its content?
Do you have NetworkManager running? which version?
It seems that Vdsm has two bugs: on boot, initscripts end up setting an
ipv6 address that Vdsm never requested.
restore-net::INFO::2016-02-25 14:14:58,024::vdsm-restore-net-config::261::root::(_find_changed_or_missing) ovirtmgmt is different or missing from persistent configuration. current: {'nic': 'enp6s0', 'dhcpv6': False, 'ipaddr': '10.80.10.117', 'mtu': '1500', 'netmask': '255.255.255.0', 'bootproto': 'none', 'stp': False, 'bridged': True, 'ipv6addr': ['2400:7d00:110:3:225:b5ff:fe00:b4f/64'], 'gateway': '10.80.10.1', 'defaultRoute': True}, persisted: {u'nic': u'enp6s0', 'dhcpv6': False, u'ipaddr': u'10.80.10.117', u'mtu': '1500', u'netmask': u'255.255.255.0', 'bootproto': 'none', 'stp': False, u'bridged': True, u'gateway': u'10.80.10.1', u'defaultRoute': True}
Then, Vdsm tries to drop the
unsolicited address, but fails. Both must be fixed ASAP.
restore-net::ERROR::2016-02-25 14:14:59,490::__init__::58::root::(__exit__) Failed rollback transaction last known good network.
Traceback (most recent call last):
File "/usr/share/vdsm/network/api.py", line 918, in setupNetworks
keep_bridge=keep_bridge)
File "/usr/share/vdsm/network/api.py", line 222, in wrapped
ret = func(**attrs)
File "/usr/share/vdsm/network/api.py", line 502, in _delNetwork
configurator.removeQoS(net_ent)
File "/usr/share/vdsm/network/configurators/__init__.py", line 122, in removeQoS
qos.remove_outbound(top_device)
File "/usr/share/vdsm/network/configurators/qos.py", line 60, in remove_outbound
device, pref=_NON_VLANNED_ID if vlan_tag is None else vlan_tag)
File "/usr/share/vdsm/network/tc/filter.py", line 31, in delete
_wrapper.process_request(command)
File "/usr/share/vdsm/network/tc/_wrapper.py", line 38, in process_request
raise TrafficControlException(retcode, err, command)
TrafficControlException: (None, 'Message truncated', ['/usr/sbin/tc', 'filter', 'del', 'dev', 'enp6s0', 'pref', '5000'])
Regards,
Dan.
_______________________________________________
Users mailing list
<a href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
</blockquote>
</body>
</html>