[ovirt-devel] [VDSM] New network test failing randomly on jenkins

[Nir Soffer]

Hi all,

I found this failure today, hopefully our network guys can look into this.

21:06:22 ======================================================================
21:06:22 ERROR: test_local_auto_with_dynamic_address_from_ra
(network.netinfo_test.TestIPv6Addresses)
21:06:22 ----------------------------------------------------------------------
21:06:22 Traceback (most recent call last):
21:06:22   File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testValidation.py",
line 97, in wrapper
21:06:22     return f(*args, **kwargs)
21:06:22   File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/network/netinfo_test.py",
line 442, in test_local_auto_with_dynamic_address_from_ra
21:06:22     self.assertEqual('link', ip_addrs[1]['scope'])
21:06:22   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
21:06:22     self.gen.next()
21:06:22   File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/network/nettestlib.py",
line 405, in dnsmasq_run
21:06:22     server.stop()
21:06:22   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
21:06:22     self.gen.next()
21:06:22   File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/network/firewall.py",
line 45, in allow_dhcp
21:06:22     _forbid_dhcp(iface)
21:06:22   File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/network/firewall.py",
line 88, in _forbid_dhcp
21:06:22     '547', '-j', 'ACCEPT'])  # DHCPv6
21:06:22   File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/network/firewall.py",
line 106, in _execCmdChecker
21:06:22     command, out, err))
21:06:22 FirewallError: Command ['/sbin/iptables', '-D', 'INPUT',
'-i', 'veth_XnInLA4lQQ', '-p', 'udp', '--sport', '546', '--dport',
'547', '-j', 'ACCEPT'] failed with []; ['Another app is currently
holding the xtables lock. Perhaps you want to use the -w option?']
21:06:22 -------------------- >> begin captured logging << --------------------
21:06:22 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip link
add name veth_XnInLA4lQQ type veth peer name veth_tbTbPAYtQz (cwd
None)
21:06:22 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0
21:06:22 root: DEBUG: /usr/bin/taskset --cpu-list 0-1
/usr/sbin/dnsmasq --dhcp-authoritative -p 0 --dhcp-option=3
--dhcp-option=6 -i veth_XnInLA4lQQ -I lo -d --bind-dynamic --enable-ra
--dhcp-range=2001:1:1:1::,slaac,2m (cwd None)
21:06:22 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/service
iptables status (cwd None)
21:06:22 root: DEBUG: SUCCESS: <err> = 'Redirecting to /bin/systemctl
status  iptables.service\nRunning in chroot, ignoring request.\n';
<rc> = 0
21:06:22 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/iptables
-I INPUT -i veth_XnInLA4lQQ -p udp --sport 68 --dport 67 -j ACCEPT
(cwd None)
21:06:22 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0
21:06:22 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/iptables
-I INPUT -i veth_XnInLA4lQQ -p udp --sport 546 --dport 547 -j ACCEPT
(cwd None)
21:06:22 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0
21:06:22 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip link
set dev veth_tbTbPAYtQz up (cwd None)
21:06:22 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0
21:06:22 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip link
set dev veth_XnInLA4lQQ up (cwd None)
21:06:22 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0
21:06:22 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip -6 addr
add dev veth_XnInLA4lQQ 2001:1:1:1::1/64 (cwd None)
21:06:22 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0
21:06:22 root: DEBUG: dnsmasq: started, version 2.75 DNS disabled
21:06:22 dnsmasq: compile time options: IPv6 GNU-getopt DBus no-i18n
IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect
inotify
21:06:22 dnsmasq: warning: interface veth_XnInLA4lQQ does not currently exist
21:06:22 dnsmasq-dhcp: router advertisement on 2001:1:1:1::
21:06:22 dnsmasq-dhcp: IPv6 router advertisement enabled
21:06:22 dnsmasq-dhcp: RTR-ADVERT(veth_XnInLA4lQQ) 2001:1:1:1::
21:06:22
21:06:22 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/service
iptables status (cwd None)
21:06:22 root: DEBUG: SUCCESS: <err> = 'Redirecting to /bin/systemctl
status  iptables.service\nRunning in chroot, ignoring request.\n';
<rc> = 0
21:06:22 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/iptables
-D INPUT -i veth_XnInLA4lQQ -p udp --sport 68 --dport 67 -j ACCEPT
(cwd None)
21:06:22 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0
21:06:23 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/iptables
-D INPUT -i veth_XnInLA4lQQ -p udp --sport 546 --dport 547 -j ACCEPT
(cwd None)
21:06:23 root: DEBUG: FAILED: <err> = 'Another app is currently
holding the xtables lock. Perhaps you want to use the -w option?\n';
<rc> = 4
21:06:23 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip link
del dev veth_XnInLA4lQQ (cwd None)
21:06:23 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0
21:06:23 --------------------- >> end captured logging << ---------------------