[VDSM] New network test failing randomly on jenkins

Hi all, I found this failure today, hopefully our network guys can look into this. 21:06:22 ====================================================================== 21:06:22 ERROR: test_local_auto_with_dynamic_address_from_ra (network.netinfo_test.TestIPv6Addresses) 21:06:22 ---------------------------------------------------------------------- 21:06:22 Traceback (most recent call last): 21:06:22 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testValidation.py", line 97, in wrapper 21:06:22 return f(*args, **kwargs) 21:06:22 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/network/netinfo_test.py", line 442, in test_local_auto_with_dynamic_address_from_ra 21:06:22 self.assertEqual('link', ip_addrs[1]['scope']) 21:06:22 File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ 21:06:22 self.gen.next() 21:06:22 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/network/nettestlib.py", line 405, in dnsmasq_run 21:06:22 server.stop() 21:06:22 File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ 21:06:22 self.gen.next() 21:06:22 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/network/firewall.py", line 45, in allow_dhcp 21:06:22 _forbid_dhcp(iface) 21:06:22 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/network/firewall.py", line 88, in _forbid_dhcp 21:06:22 '547', '-j', 'ACCEPT']) # DHCPv6 21:06:22 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/network/firewall.py", line 106, in _execCmdChecker 21:06:22 command, out, err)) 21:06:22 FirewallError: Command ['/sbin/iptables', '-D', 'INPUT', '-i', 'veth_XnInLA4lQQ', '-p', 'udp', '--sport', '546', '--dport', '547', '-j', 'ACCEPT'] failed with []; ['Another app is currently holding the xtables lock. Perhaps you want to use the -w option?'] 21:06:22 -------------------- >> begin captured logging << -------------------- 21:06:22 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip link add name veth_XnInLA4lQQ type veth peer name veth_tbTbPAYtQz (cwd None) 21:06:22 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0 21:06:22 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /usr/sbin/dnsmasq --dhcp-authoritative -p 0 --dhcp-option=3 --dhcp-option=6 -i veth_XnInLA4lQQ -I lo -d --bind-dynamic --enable-ra --dhcp-range=2001:1:1:1::,slaac,2m (cwd None) 21:06:22 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/service iptables status (cwd None) 21:06:22 root: DEBUG: SUCCESS: <err> = 'Redirecting to /bin/systemctl status iptables.service\nRunning in chroot, ignoring request.\n'; <rc> = 0 21:06:22 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/iptables -I INPUT -i veth_XnInLA4lQQ -p udp --sport 68 --dport 67 -j ACCEPT (cwd None) 21:06:22 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0 21:06:22 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/iptables -I INPUT -i veth_XnInLA4lQQ -p udp --sport 546 --dport 547 -j ACCEPT (cwd None) 21:06:22 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0 21:06:22 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip link set dev veth_tbTbPAYtQz up (cwd None) 21:06:22 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0 21:06:22 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip link set dev veth_XnInLA4lQQ up (cwd None) 21:06:22 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0 21:06:22 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip -6 addr add dev veth_XnInLA4lQQ 2001:1:1:1::1/64 (cwd None) 21:06:22 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0 21:06:22 root: DEBUG: dnsmasq: started, version 2.75 DNS disabled 21:06:22 dnsmasq: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify 21:06:22 dnsmasq: warning: interface veth_XnInLA4lQQ does not currently exist 21:06:22 dnsmasq-dhcp: router advertisement on 2001:1:1:1:: 21:06:22 dnsmasq-dhcp: IPv6 router advertisement enabled 21:06:22 dnsmasq-dhcp: RTR-ADVERT(veth_XnInLA4lQQ) 2001:1:1:1:: 21:06:22 21:06:22 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/service iptables status (cwd None) 21:06:22 root: DEBUG: SUCCESS: <err> = 'Redirecting to /bin/systemctl status iptables.service\nRunning in chroot, ignoring request.\n'; <rc> = 0 21:06:22 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/iptables -D INPUT -i veth_XnInLA4lQQ -p udp --sport 68 --dport 67 -j ACCEPT (cwd None) 21:06:22 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0 21:06:23 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/iptables -D INPUT -i veth_XnInLA4lQQ -p udp --sport 546 --dport 547 -j ACCEPT (cwd None) 21:06:23 root: DEBUG: FAILED: <err> = 'Another app is currently holding the xtables lock. Perhaps you want to use the -w option?\n'; <rc> = 4 21:06:23 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip link del dev veth_XnInLA4lQQ (cwd None) 21:06:23 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0 21:06:23 --------------------- >> end captured logging << ---------------------

On Fri, Apr 08, 2016 at 11:25:55PM +0300, Nir Soffer wrote:
Hi all,
I found this failure today, hopefully our network guys can look into this.
21:06:22 ====================================================================== 21:06:22 ERROR: test_local_auto_with_dynamic_address_from_ra (network.netinfo_test.TestIPv6Addresses) 21:06:22 ---------------------------------------------------------------------- 21:06:22 Traceback (most recent call last): 21:06:22 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testValidation.py", line 97, in wrapper 21:06:22 return f(*args, **kwargs) 21:06:22 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/network/netinfo_test.py", line 442, in test_local_auto_with_dynamic_address_from_ra 21:06:22 self.assertEqual('link', ip_addrs[1]['scope']) 21:06:22 File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ 21:06:22 self.gen.next() 21:06:22 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/network/nettestlib.py", line 405, in dnsmasq_run 21:06:22 server.stop() 21:06:22 File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ 21:06:22 self.gen.next() 21:06:22 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/network/firewall.py", line 45, in allow_dhcp 21:06:22 _forbid_dhcp(iface) 21:06:22 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/network/firewall.py", line 88, in _forbid_dhcp 21:06:22 '547', '-j', 'ACCEPT']) # DHCPv6 21:06:22 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/network/firewall.py", line 106, in _execCmdChecker 21:06:22 command, out, err)) 21:06:22 FirewallError: Command ['/sbin/iptables', '-D', 'INPUT', '-i', 'veth_XnInLA4lQQ', '-p', 'udp', '--sport', '546', '--dport', '547', '-j', 'ACCEPT'] failed with []; ['Another app is currently holding the xtables lock. Perhaps you want to use the -w option?']
Thanks Nir. I hope that https://gerrit.ovirt.org/55904 would solve the issue.
participants (2)
-
Dan Kenigsberg
-
Nir Soffer