Re: [ovirt-devel] [VDSM] Failing networking tests

Hi all, These tests seems to fail unrelated builds: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/1600/con... Seen it once, please check. 19:43:05 ====================================================================== 19:43:05 ERROR: test_local_auto_with_dynamic_address_from_ra (network.netinfo_test.TestIPv6Addresses) 19:43:05 ---------------------------------------------------------------------- 19:43:05 Traceback (most recent call last): 19:43:05 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/tests/testValidation.py", line 97, in wrapper 19:43:05 return f(*args, **kwargs) 19:43:05 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/tests/network/netinfo_test.py", line 419, in test_local_auto_with_dynamic_address_from_ra 19:43:05 IPV6_NETPREFIX_LEN, family=6) 19:43:05 File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ 19:43:05 self.gen.next() 19:43:05 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/tests/network/nettestlib.py", line 454, in wait_for_ipv6 19:43:05 raise Exception('IPv6 addresses has not been caught within the ' 19:43:05 Exception: IPv6 addresses has not been caught within the given timeout. 19:43:05 19:43:05 -------------------- >> begin captured logging << -------------------- 19:43:05 2016-09-04 19:42:32,949 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-1 /sbin/ip link add name veth_hU8rsgT1oy type veth peer name veth_OgkZCPxdBp (cwd None) 19:43:05 2016-09-04 19:42:32,962 DEBUG [root] (MainThread) SUCCESS: <err> = ''; <rc> = 0 19:43:05 2016-09-04 19:42:32,965 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-1 /usr/sbin/dnsmasq --dhcp-authoritative -p 0 --dhcp-option=3 --dhcp-option=6 -i veth_hU8rsgT1oy -I lo -d --bind-dynamic --enable-ra --dhcp-range=2001:1:1:1::,slaac,2m (cwd None) 19:43:05 2016-09-04 19:42:33,475 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-1 /sbin/service iptables status (cwd None) 19:43:05 2016-09-04 19:42:33,501 DEBUG [root] (MainThread) SUCCESS: <err> = 'Redirecting to /bin/systemctl status iptables.service\nRunning in chroot, ignoring request.\n'; <rc> = 0 19:43:05 2016-09-04 19:42:33,502 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-1 /sbin/iptables --wait -I INPUT -i veth_hU8rsgT1oy -p udp --sport 68 --dport 67 -j ACCEPT (cwd None) 19:43:05 2016-09-04 19:42:33,514 DEBUG [root] (MainThread) SUCCESS: <err> = ''; <rc> = 0 19:43:05 2016-09-04 19:42:33,515 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-1 /sbin/iptables --wait -I INPUT -i veth_hU8rsgT1oy -p udp --sport 546 --dport 547 -j ACCEPT (cwd None) 19:43:05 2016-09-04 19:42:33,523 DEBUG [root] (MainThread) SUCCESS: <err> = ''; <rc> = 0 19:43:05 2016-09-04 19:42:33,525 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-1 /sbin/ip link set dev veth_OgkZCPxdBp up (cwd None) 19:43:05 2016-09-04 19:42:33,532 DEBUG [root] (MainThread) SUCCESS: <err> = ''; <rc> = 0 19:43:05 2016-09-04 19:42:33,533 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-1 /sbin/ip link set dev veth_hU8rsgT1oy up (cwd None) 19:43:05 2016-09-04 19:42:33,543 DEBUG [root] (MainThread) SUCCESS: <err> = ''; <rc> = 0 19:43:05 2016-09-04 19:42:33,543 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-1 /sbin/ip -6 addr add dev veth_hU8rsgT1oy 2001:1:1:1::1/64 (cwd None) 19:43:05 2016-09-04 19:42:33,550 DEBUG [root] (MainThread) SUCCESS: <err> = ''; <rc> = 0 19:43:05 2016-09-04 19:42:53,541 DEBUG [root] (MainThread) dnsmasq: started, version 2.76 DNS disabled 19:43:05 dnsmasq: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify 19:43:05 dnsmasq: warning: interface veth_hU8rsgT1oy does not currently exist 19:43:05 dnsmasq-dhcp: router advertisement on 2001:1:1:1:: 19:43:05 dnsmasq-dhcp: IPv6 router advertisement enabled 19:43:05 dnsmasq-dhcp: RTR-ADVERT(veth_hU8rsgT1oy) 2001:1:1:1:: 19:43:05 dnsmasq-dhcp: RTR-ADVERT(veth_hU8rsgT1oy) 2001:1:1:1:: 19:43:05 19:43:05 2016-09-04 19:42:53,542 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-1 /sbin/service iptables status (cwd None) 19:43:05 2016-09-04 19:42:53,589 DEBUG [root] (MainThread) SUCCESS: <err> = 'Redirecting to /bin/systemctl status iptables.service\nRunning in chroot, ignoring request.\n'; <rc> = 0 19:43:05 2016-09-04 19:42:53,590 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-1 /sbin/iptables --wait -D INPUT -i veth_hU8rsgT1oy -p udp --sport 68 --dport 67 -j ACCEPT (cwd None) 19:43:05 2016-09-04 19:42:53,612 DEBUG [root] (MainThread) SUCCESS: <err> = ''; <rc> = 0 19:43:05 2016-09-04 19:42:53,613 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-1 /sbin/iptables --wait -D INPUT -i veth_hU8rsgT1oy -p udp --sport 546 --dport 547 -j ACCEPT (cwd None) 19:43:05 2016-09-04 19:42:53,630 DEBUG [root] (MainThread) SUCCESS: <err> = ''; <rc> = 0 19:43:05 2016-09-04 19:42:53,631 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-1 /sbin/ip link del dev veth_hU8rsgT1oy (cwd None) 19:43:05 2016-09-04 19:42:53,669 DEBUG [root] (MainThread) SUCCESS: <err> = ''; <rc> = 0 19:43:05 --------------------- >> end captured logging << --------------------- 19:43:05 19:43:05 ====================================================================== 19:43:05 FAIL: test_events_keys (network.netlink_test.NetlinkEventMonitorTests) 19:43:05 ---------------------------------------------------------------------- 19:43:05 Traceback (most recent call last): 19:43:05 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/tests/testValidation.py", line 97, in wrapper 19:43:05 return f(*args, **kwargs) 19:43:05 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/tests/network/netlink_test.py", line 173, in test_events_keys 19:43:05 for d in _caught]))) 19:43:05 AssertionError: Expected events have not been caught (in the right order). 19:43:05 Expected: 19:43:05 {'event': 'new_link', 'name': 'dummy_40rKa'} 19:43:05 {'event': 'new_addr', 'address': '192.0.2.1/24'} 19:43:05 {'event': 'new_link', 'name': 'dummy_40rKa'} 19:43:05 {'event': 'new_addr', 'family': 'inet6'} 19:43:05 {'event': 'del_neigh'} 19:43:05 {'event': 'del_addr', 'family': 'inet6'} 19:43:05 {'event': 'del_addr', 'address': '192.0.2.1/24'} 19:43:05 {'destination': '192.0.2.1', 'event': 'del_route'} 19:43:05 {'event': 'del_link', 'name': 'dummy_40rKa'}. 19:43:05 Caught: 19:43:05 {'event': 'new_link', 'name': 'dummy_40rKa', 'address': '6e:f3:b9:80:5d:20'} 19:43:05 {'event': 'new_addr', 'family': 'inet', 'address': '192.0.2.1/24'} 19:43:05 {'destination': '192.0.2.1', 'event': 'new_route', 'family': 'inet'} 19:43:05 {'event': 'new_link', 'name': 'dummy_40rKa', 'address': '6e:f3:b9:80:5d:20'} 19:43:05 {'destination': '192.0.2.0/24', 'event': 'new_route', 'family': 'inet'} 19:43:05 {'destination': '192.0.2.0', 'event': 'new_route', 'family': 'inet'} 19:43:05 {'destination': '192.0.2.255', 'event': 'new_route', 'family': 'inet'} 19:43:05 {'destination': 'ff00::/8', 'event': 'new_route', 'family': 'inet6'} 19:43:05 {'destination': 'fe80::/64', 'event': 'new_route', 'family': 'inet6'} 19:43:05 {'event': 'new_link', 'name': 'dummy_40rKa', 'address': '6e:f3:b9:80:5d:20'} 19:43:05 {'event': 'new_addr', 'family': 'inet6', 'address': 'fe80::6cf3:b9ff:fe80:5d20/64'} 19:43:05 {'destination': 'fe80::6cf3:b9ff:fe80:5d20', 'event': 'new_route', 'family': 'inet6'} 19:43:05 {'destination': 'fe80::', 'event': 'new_route', 'family': 'inet6'} 19:43:05 {'event': 'new_link', 'name': 'dummy_40rKa', 'address': '6e:f3:b9:80:5d:20'} 19:43:05 {'destination': 'fe80::/64', 'event': 'del_route', 'family': 'inet6'} 19:43:05 {'destination': 'ff00::/8', 'event': 'del_route', 'family': 'inet6'} 19:43:05 {'event': 'del_addr', 'family': 'inet6', 'address': 'fe80::6cf3:b9ff:fe80:5d20/64'} 19:43:05 {'destination': 'fe80::', 'event': 'del_route', 'family': 'inet6'} 19:43:05 {'destination': 'fe80::6cf3:b9ff:fe80:5d20', 'event': 'del_route', 'family': 'inet6'} 19:43:05 {'event': 'del_addr', 'family': 'inet', 'address': '192.0.2.1/24'} 19:43:05 {'destination': '192.0.2.1', 'event': 'del_route', 'family': 'inet'} 19:43:05 {'event': 'del_link', 'name': 'dummy_40rKa', 'address': '6e:f3:b9:80:5d:20'}. 19:43:05 -------------------- >> begin captured logging << -------------------- 19:43:05 2016-09-04 19:42:54,605 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-1 /sbin/ip link add name dummy_40rKa type dummy (cwd None) 19:43:05 2016-09-04 19:42:54,623 DEBUG [root] (MainThread) SUCCESS: <err> = ''; <rc> = 0 19:43:05 2016-09-04 19:42:54,623 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-1 /sbin/ip -4 addr add dev dummy_40rKa 192.0.2.1/24 (cwd None) 19:43:05 2016-09-04 19:42:54,640 DEBUG [root] (MainThread) SUCCESS: <err> = ''; <rc> = 0 19:43:05 2016-09-04 19:42:54,642 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-1 /sbin/ip link set dev dummy_40rKa up (cwd None) 19:43:05 2016-09-04 19:42:54,658 DEBUG [root] (MainThread) SUCCESS: <err> = ''; <rc> = 0 19:43:16 2016-09-04 19:42:54,659 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-1 /sbin/ip link del dev dummy_40rKa (cwd None) 19:43:16 2016-09-04 19:42:54,684 DEBUG [root] (MainThread) SUCCESS: <err> = ''; <rc> = 0 19:43:16 --------------------- >> end captured logging << ---------------------