<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Thu, Mar 3, 2016 at 2:54 AM, David LeVene <span dir="ltr"><<a href="mailto:David.LeVene@blackboard.com" target="_blank">David.LeVene@blackboard.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
Thanks for the quick responses & help.. answers in-line at the end of this email.<br>
<br>
Cheers<br>
David<br>
<div><div class="h5"><br>
-----Original Message-----<br>
From: Edward Haas [mailto:<a href="mailto:edwardh@redhat.com">edwardh@redhat.com</a>]<br>
Sent: Wednesday, March 02, 2016 20:05<br>
To: David LeVene <<a href="mailto:David.LeVene@blackboard.com">David.LeVene@blackboard.com</a>>; Dan Kenigsberg <<a href="mailto:danken@redhat.com">danken@redhat.com</a>><br>
Cc: <a href="mailto:users@ovirt.org">users@ovirt.org</a><br>
Subject: Re: [ovirt-users] 3.6 looses network on reboot<br>
<br>
On 03/02/2016 01:36 AM, David LeVene wrote:<br>
> Hi Dan,<br>
><br>
> I missed the email as the subject line changed!<br>
><br>
> So we use and run IPv6 in our network - not sure if this is related. The Addresses are handed out via SLAAC so that would be where the IPv6 address is coming from.<br>
><br>
> My memory is a bit sketchy... but I think if I remove the vmfex/SRIOV vNIC and only run with the one vNIC it works fine, it's when I bring the second NIC into play with SRIOV the issues arise.<br>
><br>
> Answers inline.<br>
><br>
> -----Original Message-----<br>
> From: Dan Kenigsberg [mailto:<a href="mailto:danken@redhat.com">danken@redhat.com</a>]<br>
> Sent: Tuesday, March 01, 2016 00:28<br>
> To: David LeVene <<a href="mailto:David.LeVene@blackboard.com">David.LeVene@blackboard.com</a>><br>
> Cc: <a href="mailto:edwardh@redhat.com">edwardh@redhat.com</a>; <a href="mailto:users@ovirt.org">users@ovirt.org</a><br>
> Subject: Re: [ovirt-users] 3.6 looses network on reboot<br>
><br>
> This sounds very bad. Changing the subject, so the wider, more problematic issue is visible.<br>
><br>
> Did any other user see this behavior?<br>
><br>
> On Mon, Feb 29, 2016 at 06:27:46AM +0000, David LeVene wrote:<br>
>> Hi Dan,<br>
>><br>
>> Answers as follows;<br>
>><br>
>> # rpm -qa | grep -i vdsm<br>
>> vdsm-jsonrpc-4.17.18-1.el7.noarch<br>
>> vdsm-hook-vmfex-4.17.18-1.el7.noarch<br>
>> vdsm-infra-4.17.18-1.el7.noarch<br>
>> vdsm-4.17.18-1.el7.noarch<br>
>> vdsm-python-4.17.18-1.el7.noarch<br>
>> vdsm-yajsonrpc-4.17.18-1.el7.noarch<br>
>> vdsm-cli-4.17.18-1.el7.noarch<br>
>> vdsm-xmlrpc-4.17.18-1.el7.noarch<br>
>> vdsm-hook-vmfex-dev-4.17.18-1.el7.noarch<br>
>><br>
>><br>
>> There was in this folder ifcfg-ovirtmgnt bridge setup, and also route-ovirtmgnt & rule-ovirtmgmt.. but they were removed after the reboot.<br>
>><br>
>> # ls -althr | grep ifcfg<br>
>> -rw-r--r--. 1 root root 254 Sep 16 21:21 ifcfg-lo -rw-r--r--. 1 root<br>
>> root 120 Feb 25 14:07 ifcfg-enp7s0f0 -rw-rw-r--. 1 root root 174<br>
>> Feb<br>
>> 25 14:40 ifcfg-enp6s0<br>
>><br>
>> I think I modified ifcfg-enp6s0 to get networking up again (eg was set to bridge.. but the bridge wasn't configured).. it was a few days ago.. if it's important I can reboot the box again to see what state it comes up with.<br>
>><br>
>> # cat ifcfg-enp6s0<br>
>> BOOTPROTO="none"<br>
>> IPADDR="10.80.10.117"<br>
>> NETMASK="255.255.255.0"<br>
>> GATEWAY="10.80.10.1"<br>
>> DEVICE="enp6s0"<br>
>> HWADDR="00:25:b5:00:0b:4f"<br>
>> ONBOOT=yes<br>
>> PEERDNS=yes<br>
>> PEERROUTES=yes<br>
>> MTU=1500<br>
>><br>
>> # cat ifcfg-enp7s0f0<br>
>> # Generated by VDSM version 4.17.18-1.el7<br>
>> DEVICE=enp7s0f0<br>
>> ONBOOT=yes<br>
>> MTU=1500<br>
>> HWADDR=00:25:b5:00:0b:0f<br>
>> NM_CONTROLLED=no<br>
>><br>
>> # find /var/lib/vdsm/persistence<br>
>> /var/lib/vdsm/persistence<br>
>> /var/lib/vdsm/persistence/netconf<br>
>> /var/lib/vdsm/persistence/netconf.1456371473833165545<br>
>> /var/lib/vdsm/persistence/netconf.1456371473833165545/nets<br>
>> /var/lib/vdsm/persistence/netconf.1456371473833165545/nets/ovirtmgmt<br>
>><br>
>> # cat<br>
>> /var/lib/vdsm/persistence/netconf.1456371473833165545/nets/ovirtmgmt<br>
>> {<br>
>> "nic": "enp6s0",<br>
>> "ipaddr": "10.80.10.117",<br>
>> "mtu": "1500",<br>
>> "netmask": "255.255.255.0",<br>
>> "STP": "no",<br>
>> "bridged": "true",<br>
>> "gateway": "10.80.10.1",<br>
>> "defaultRoute": true<br>
>> }<br>
>><br>
>> Supervdsm log is attached.<br>
><br>
> Have you editted ifcfg-ovirtmgmt manually?<br>
> Nope<br>
><br>
> Can you somehow reproduce it, and share its content?<br>
> Yea, I should be able to reproduce it - just gotta fix it first (create the networking manually and get VDSM on-line). Also it’s a side project/investigation at the moment so time isn't on my side...<br>
><br>
> Would it help if I take an sosreport before and after? I don’t' mine emailing these directly to yourself.<br>
><br>
> Do you have NetworkManager running? which version?<br>
> NM is disabled, but the version is...<br>
> # rpm -q NetworkManager<br>
> NetworkManager-1.0.6-27.el7.x86_64<br>
> # systemctl status NetworkManager.service ● NetworkManager.service -<br>
> Network Manager<br>
> Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; disabled; vendor preset: enabled)<br>
> Active: inactive (dead)<br>
><br>
> It seems that Vdsm has two bugs: on boot, initscripts end up setting<br>
> an<br>
> ipv6 address that Vdsm never requested<br>
><br>
> As mentioned above this would have come from SLAAC which we have setup<br>
> in our network<br>
><br>
> restore-net::INFO::2016-02-25<br>
> 14:14:58,024::vdsm-restore-net-config::261::root::(_find_changed_or_mi<br>
> ssing) ovirtmgmt is different or missing from persistent<br>
> configuration. current: {'nic': 'enp6s0', 'dhcpv6': False, 'ipaddr':<br>
> '10.80.10.117', 'mtu': '1500', 'netmask': '255.255.255.0',<br>
> 'bootproto': 'none', 'stp': False, 'bridged': True, 'ipv6addr':<br>
> ['2400:7d00:110:3:225:b5ff:fe00:b4f/64'], 'gateway': '10.80.10.1',<br>
> 'defaultRoute': True}, persisted: {u'nic': u'enp6s0', 'dhcpv6': False,<br>
> u'ipaddr': u'10.80.10.117', u'mtu': '1500', u'netmask':<br>
> u'255.255.255.0', 'bootproto': 'none', 'stp': False, u'bridged': True,<br>
> u'gateway': u'10.80.10.1', u'defaultRoute': True}<br>
><br>
><br>
> Then, Vdsm tries to drop the<br>
> unsolicited address, but fails. Both must be fixed ASAP.<br>
><br>
> restore-net::ERROR::2016-02-25 14:14:59,490::__init__::58::root::(__exit__) Failed rollback transaction last known good network.<br>
> Traceback (most recent call last):<br>
> File "/usr/share/vdsm/network/api.py", line 918, in setupNetworks<br>
> keep_bridge=keep_bridge)<br>
> File "/usr/share/vdsm/network/api.py", line 222, in wrapped<br>
> ret = func(**attrs)<br>
> File "/usr/share/vdsm/network/api.py", line 502, in _delNetwork<br>
> configurator.removeQoS(net_ent)<br>
> File "/usr/share/vdsm/network/configurators/__init__.py", line 122, in removeQoS<br>
> qos.remove_outbound(top_device)<br>
> File "/usr/share/vdsm/network/configurators/qos.py", line 60, in remove_outbound<br>
> device, pref=_NON_VLANNED_ID if vlan_tag is None else vlan_tag)<br>
> File "/usr/share/vdsm/network/tc/filter.py", line 31, in delete<br>
> _wrapper.process_request(command)<br>
> File "/usr/share/vdsm/network/tc/_wrapper.py", line 38, in process_request<br>
> raise TrafficControlException(retcode, err, command)<br>
> TrafficControlException: (None, 'Message truncated',<br>
> ['/usr/sbin/tc', 'filter', 'del', 'dev', 'enp6s0', 'pref', '5000'])<br>
><br>
> Regards,<br>
> Dan.<br>
><br>
<br>
Hi David,<br>
<br>
You have encountered two issues, the first with IPv6, which we do not fully support in 3.6 and a the second with an unmanaged failure during network setup on boot.<br>
We are going to back-port both fixes very soon.<br>
<br>
Can you check our patches? They should resolve the problem we saw in the<br>
log: <a href="https://gerrit.ovirt.org/#/c/54237" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/54237</a> (based on oVirt-3.6.3)<br>
<br>
</div></div>-- I've manually applied the patch to the node that I was testing on and the networking comes on-line correctly - now I'm encountering a gluster issue with cannot find master domain.<br></blockquote><div><br></div><div>Please attach vdsm logs showing gluster connection attempts.</div><div><br></div><div>You should have also interesting logs in /var/log/glusterfs/ - there should be a log for each</div><div>gluster connection (server:/path). </div><div><br></div><div>Nir</div></div></div></div>