[VDSM] New network test failure - ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type
Failed only one patch when building 4, so this looks like a random failure, but the error is disturbing. See http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2255/consol... 17:00:31 ====================================================================== 17:00:31 ERROR: test_bond_create_failure_on_slave_add (network.link_bond_test.LinkBondTests) 17:00:31 ---------------------------------------------------------------------- 17:00:31 Traceback (most recent call last): 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/tests/network/link_bond_test.py", line 83, in test_bond_create_failure_on_slave_add 17:00:31 base_bond.add_slaves((nic1, nic2)) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py", line 153, in add_slaves 17:00:31 with _preserve_iface_state(slave): 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 17:00:31 return next(self.gen) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py", line 214, in _preserve_iface_state 17:00:31 dev_was_up = iface.is_up(dev) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py", line 54, in is_up 17:00:31 return is_admin_up(dev) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py", line 58, in is_admin_up 17:00:31 return is_link_up(get_link(dev)['flags'], check_oper_status=False) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py", line 56, in get_link 17:00:31 with _get_link(name=name, sock=sock) as link: 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 17:00:31 return next(self.gen) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py", line 199, in _get_link 17:00:31 err = _rtnl_link_get_kernel(sock, index, name, byref(link)) 17:00:31 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type 17:00:31 -------------------- >> begin captured logging << -------------------- 17:00:31 2016-09-20 17:00:12,586 WARNING [py.warnings] (MainThread) /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead 17:00:31 inspect.getargspec(func) 17:00:31 (inspect:1041) 17:00:31 2016-09-20 17:00:12,587 INFO [root] (MainThread) Bond check_7lzvb has been created. (bond:142) 17:00:31 2016-09-20 17:00:12,598 INFO [root] (MainThread) Bond check_7lzvb has been destroyed. (bond:149) 17:00:31 2016-09-20 17:00:12,598 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_Dmwc5 type dummy (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,607 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,608 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_xIK5C type dummy (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,615 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,617 INFO [root] (MainThread) Bond bond_rwTTFu has been created. (bond:142) 17:00:31 2016-09-20 17:00:12,634 INFO [root] (MainThread) Bond bond_rwTTFu has been destroyed. (bond:149) 17:00:31 2016-09-20 17:00:12,634 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_Dmwc5 (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,658 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,659 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_xIK5C (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,676 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 --------------------- >> end captured logging << ---------------------
On Tue, Sep 20, 2016 at 8:18 PM, Nir Soffer <nsoffer@redhat.com> wrote:
Failed only one patch when building 4, so this looks like a random failure, but the error is disturbing.
See http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2255/consol...
17:00:31 ====================================================================== 17:00:31 ERROR: test_bond_create_failure_on_slave_add (network.link_bond_test.LinkBondTests) 17:00:31 ---------------------------------------------------------------------- 17:00:31 Traceback (most recent call last): 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/tests/network/link_bond_test.py", line 83, in test_bond_create_failure_on_slave_add 17:00:31 base_bond.add_slaves((nic1, nic2)) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py", line 153, in add_slaves 17:00:31 with _preserve_iface_state(slave): 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 17:00:31 return next(self.gen) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py", line 214, in _preserve_iface_state 17:00:31 dev_was_up = iface.is_up(dev) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py", line 54, in is_up 17:00:31 return is_admin_up(dev) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py", line 58, in is_admin_up 17:00:31 return is_link_up(get_link(dev)['flags'], check_oper_status=False) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py", line 56, in get_link 17:00:31 with _get_link(name=name, sock=sock) as link: 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 17:00:31 return next(self.gen) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py", line 199, in _get_link 17:00:31 err = _rtnl_link_get_kernel(sock, index, name, byref(link)) 17:00:31 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type 17:00:31 -------------------- >> begin captured logging << -------------------- 17:00:31 2016-09-20 17:00:12,586 WARNING [py.warnings] (MainThread) /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead 17:00:31 inspect.getargspec(func) 17:00:31 (inspect:1041) 17:00:31 2016-09-20 17:00:12,587 INFO [root] (MainThread) Bond check_7lzvb has been created. (bond:142) 17:00:31 2016-09-20 17:00:12,598 INFO [root] (MainThread) Bond check_7lzvb has been destroyed. (bond:149) 17:00:31 2016-09-20 17:00:12,598 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_Dmwc5 type dummy (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,607 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,608 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_xIK5C type dummy (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,615 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,617 INFO [root] (MainThread) Bond bond_rwTTFu has been created. (bond:142) 17:00:31 2016-09-20 17:00:12,634 INFO [root] (MainThread) Bond bond_rwTTFu has been destroyed. (bond:149) 17:00:31 2016-09-20 17:00:12,634 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_Dmwc5 (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,658 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,659 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_xIK5C (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,676 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 --------------------- >> end captured logging << ---------------------
Another instance, again only one patch failed when building 5 patches series. This time with new log format for the tests. 18:56:15 ====================================================================== 18:56:15 ERROR: test_bond_create_failure_on_slave_add (network.link_bond_test.LinkBondTests) 18:56:15 ---------------------------------------------------------------------- 18:56:15 Traceback (most recent call last): 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/tests/network/link_bond_test.py", line 83, in test_bond_create_failure_on_slave_add 18:56:15 base_bond.add_slaves((nic1, nic2)) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py", line 153, in add_slaves 18:56:15 with _preserve_iface_state(slave): 18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 18:56:15 return next(self.gen) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py", line 214, in _preserve_iface_state 18:56:15 dev_was_up = iface.is_up(dev) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py", line 54, in is_up 18:56:15 return is_admin_up(dev) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py", line 58, in is_admin_up 18:56:15 return is_link_up(get_link(dev)['flags'], check_oper_status=False) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py", line 56, in get_link 18:56:15 with _get_link(name=name, sock=sock) as link: 18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 18:56:15 return next(self.gen) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py", line 199, in _get_link 18:56:15 err = _rtnl_link_get_kernel(sock, index, name, byref(link)) 18:56:15 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type 18:56:15 -------------------- >> begin captured logging << -------------------- 18:56:15 2016-09-20 18:55:55,220 WARNING [py.warnings] (MainThread) /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead 18:56:15 inspect.getargspec(func) 18:56:15 (inspect:1041) 18:56:15 2016-09-20 18:55:55,222 INFO [root] (MainThread) Bond check_pkopm has been created. (bond:142) 18:56:15 2016-09-20 18:55:55,233 INFO [root] (MainThread) Bond check_pkopm has been destroyed. (bond:149) 18:56:15 2016-09-20 18:55:55,234 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_gmayN type dummy (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_p3yZR type dummy (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,250 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 2016-09-20 18:55:55,258 INFO [root] (MainThread) Bond bond_Siiakr has been created. (bond:142) 18:56:15 2016-09-20 18:55:55,265 INFO [root] (MainThread) Bond bond_Siiakr has been destroyed. (bond:149) 18:56:15 2016-09-20 18:55:55,265 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_gmayN (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,277 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 2016-09-20 18:55:55,278 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_p3yZR (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,289 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 --------------------- >> end captured logging << ---------------------
On Wed, Sep 21, 2016 at 12:18:37AM +0300, Nir Soffer wrote:
On Tue, Sep 20, 2016 at 8:18 PM, Nir Soffer <nsoffer@redhat.com> wrote:
Failed only one patch when building 4, so this looks like a random failure, but the error is disturbing.
Petr, could you look into it? Could it be that libnl3 changed their API? Or that a slave is pulling an ancient lib? Or broken kernel? Nothing makes sense.
See http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2255/consol...
17:00:31 ====================================================================== 17:00:31 ERROR: test_bond_create_failure_on_slave_add (network.link_bond_test.LinkBondTests) 17:00:31 ---------------------------------------------------------------------- 17:00:31 Traceback (most recent call last): 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/tests/network/link_bond_test.py", line 83, in test_bond_create_failure_on_slave_add 17:00:31 base_bond.add_slaves((nic1, nic2)) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py", line 153, in add_slaves 17:00:31 with _preserve_iface_state(slave): 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 17:00:31 return next(self.gen) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py", line 214, in _preserve_iface_state 17:00:31 dev_was_up = iface.is_up(dev) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py", line 54, in is_up 17:00:31 return is_admin_up(dev) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py", line 58, in is_admin_up 17:00:31 return is_link_up(get_link(dev)['flags'], check_oper_status=False) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py", line 56, in get_link 17:00:31 with _get_link(name=name, sock=sock) as link: 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 17:00:31 return next(self.gen) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py", line 199, in _get_link 17:00:31 err = _rtnl_link_get_kernel(sock, index, name, byref(link)) 17:00:31 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type 17:00:31 -------------------- >> begin captured logging << -------------------- 17:00:31 2016-09-20 17:00:12,586 WARNING [py.warnings] (MainThread) /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead 17:00:31 inspect.getargspec(func) 17:00:31 (inspect:1041)
Piotr, the DeprecationWarning is unrelated to our code but we are using this API, too.
17:00:31 2016-09-20 17:00:12,587 INFO [root] (MainThread) Bond check_7lzvb has been created. (bond:142) 17:00:31 2016-09-20 17:00:12,598 INFO [root] (MainThread) Bond check_7lzvb has been destroyed. (bond:149) 17:00:31 2016-09-20 17:00:12,598 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_Dmwc5 type dummy (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,607 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,608 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_xIK5C type dummy (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,615 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,617 INFO [root] (MainThread) Bond bond_rwTTFu has been created. (bond:142) 17:00:31 2016-09-20 17:00:12,634 INFO [root] (MainThread) Bond bond_rwTTFu has been destroyed. (bond:149) 17:00:31 2016-09-20 17:00:12,634 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_Dmwc5 (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,658 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,659 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_xIK5C (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,676 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 --------------------- >> end captured logging << ---------------------
Another instance, again only one patch failed when building 5 patches series.
This time with new log format for the tests.
18:56:15 ====================================================================== 18:56:15 ERROR: test_bond_create_failure_on_slave_add (network.link_bond_test.LinkBondTests) 18:56:15 ---------------------------------------------------------------------- 18:56:15 Traceback (most recent call last): 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/tests/network/link_bond_test.py", line 83, in test_bond_create_failure_on_slave_add 18:56:15 base_bond.add_slaves((nic1, nic2)) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py", line 153, in add_slaves 18:56:15 with _preserve_iface_state(slave): 18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 18:56:15 return next(self.gen) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py", line 214, in _preserve_iface_state 18:56:15 dev_was_up = iface.is_up(dev) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py", line 54, in is_up 18:56:15 return is_admin_up(dev) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py", line 58, in is_admin_up 18:56:15 return is_link_up(get_link(dev)['flags'], check_oper_status=False) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py", line 56, in get_link 18:56:15 with _get_link(name=name, sock=sock) as link: 18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 18:56:15 return next(self.gen) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py", line 199, in _get_link 18:56:15 err = _rtnl_link_get_kernel(sock, index, name, byref(link)) 18:56:15 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type 18:56:15 -------------------- >> begin captured logging << -------------------- 18:56:15 2016-09-20 18:55:55,220 WARNING [py.warnings] (MainThread) /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead 18:56:15 inspect.getargspec(func) 18:56:15 (inspect:1041) 18:56:15 2016-09-20 18:55:55,222 INFO [root] (MainThread) Bond check_pkopm has been created. (bond:142) 18:56:15 2016-09-20 18:55:55,233 INFO [root] (MainThread) Bond check_pkopm has been destroyed. (bond:149) 18:56:15 2016-09-20 18:55:55,234 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_gmayN type dummy (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_p3yZR type dummy (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,250 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 2016-09-20 18:55:55,258 INFO [root] (MainThread) Bond bond_Siiakr has been created. (bond:142) 18:56:15 2016-09-20 18:55:55,265 INFO [root] (MainThread) Bond bond_Siiakr has been destroyed. (bond:149) 18:56:15 2016-09-20 18:55:55,265 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_gmayN (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,277 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 2016-09-20 18:55:55,278 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_p3yZR (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,289 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 --------------------- >> end captured logging << ---------------------
On Thu, Sep 22, 2016 at 9:58 AM, Dan Kenigsberg <danken@redhat.com> wrote:
On Wed, Sep 21, 2016 at 12:18:37AM +0300, Nir Soffer wrote:
On Tue, Sep 20, 2016 at 8:18 PM, Nir Soffer <nsoffer@redhat.com> wrote:
Failed only one patch when building 4, so this looks like a random failure, but the error is disturbing.
Petr, could you look into it? Could it be that libnl3 changed their API? Or that a slave is pulling an ancient lib? Or broken kernel? Nothing makes sense.
See http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-
x86_64/2255/console
17:00:31 ============================================================
==========
17:00:31 ERROR: test_bond_create_failure_on_slave_add (network.link_bond_test.LinkBondTests) 17:00:31 ------------------------------------------------------------
17:00:31 Traceback (most recent call last): 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_ 64/vdsm/tests/network/link_bond_test.py", line 83, in test_bond_create_failure_on_slave_add 17:00:31 base_bond.add_slaves((nic1, nic2)) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_ 64/vdsm/lib/vdsm/network/link/bond.py", line 153, in add_slaves 17:00:31 with _preserve_iface_state(slave): 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 17:00:31 return next(self.gen) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_ 64/vdsm/lib/vdsm/network/link/bond.py", line 214, in _preserve_iface_state 17:00:31 dev_was_up = iface.is_up(dev) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_ 64/vdsm/lib/vdsm/network/link/iface.py", line 54, in is_up 17:00:31 return is_admin_up(dev) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_ 64/vdsm/lib/vdsm/network/link/iface.py", line 58, in is_admin_up 17:00:31 return is_link_up(get_link(dev)['flags'], check_oper_status=False) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_ 64/vdsm/lib/vdsm/network/netlink/link.py", line 56, in get_link 17:00:31 with _get_link(name=name, sock=sock) as link: 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 17:00:31 return next(self.gen) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_ 64/vdsm/lib/vdsm/network/netlink/link.py", line 199, in _get_link 17:00:31 err = _rtnl_link_get_kernel(sock, index, name, byref(link)) 17:00:31 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type 17:00:31 -------------------- >> begin captured logging <<
17:00:31 2016-09-20 17:00:12,586 WARNING [py.warnings] (MainThread) /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead 17:00:31 inspect.getargspec(func) 17:00:31 (inspect:1041)
Piotr, the DeprecationWarning is unrelated to our code but we are using this API, too.
Please take a look at https://gerrit.ovirt.org/64313
17:00:31 2016-09-20 17:00:12,587 INFO [root] (MainThread) Bond check_7lzvb has been created. (bond:142) 17:00:31 2016-09-20 17:00:12,598 INFO [root] (MainThread) Bond check_7lzvb has been destroyed. (bond:149) 17:00:31 2016-09-20 17:00:12,598 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_Dmwc5 type dummy (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,607 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,608 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_xIK5C type dummy (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,615 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,617 INFO [root] (MainThread) Bond bond_rwTTFu has been created. (bond:142) 17:00:31 2016-09-20 17:00:12,634 INFO [root] (MainThread) Bond bond_rwTTFu has been destroyed. (bond:149) 17:00:31 2016-09-20 17:00:12,634 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_Dmwc5 (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,658 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,659 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_xIK5C (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,676 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 --------------------- >> end captured logging <<
Another instance, again only one patch failed when building 5 patches
series.
This time with new log format for the tests.
18:56:15 ============================================================
==========
18:56:15 ERROR: test_bond_create_failure_on_slave_add (network.link_bond_test.LinkBondTests) 18:56:15 ------------------------------------------------------------
18:56:15 Traceback (most recent call last): 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_ 64/vdsm/tests/network/link_bond_test.py", line 83, in test_bond_create_failure_on_slave_add 18:56:15 base_bond.add_slaves((nic1, nic2)) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_ 64/vdsm/lib/vdsm/network/link/bond.py", line 153, in add_slaves 18:56:15 with _preserve_iface_state(slave): 18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 18:56:15 return next(self.gen) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_ 64/vdsm/lib/vdsm/network/link/bond.py", line 214, in _preserve_iface_state 18:56:15 dev_was_up = iface.is_up(dev) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_ 64/vdsm/lib/vdsm/network/link/iface.py", line 54, in is_up 18:56:15 return is_admin_up(dev) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_ 64/vdsm/lib/vdsm/network/link/iface.py", line 58, in is_admin_up 18:56:15 return is_link_up(get_link(dev)['flags'], check_oper_status=False) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_ 64/vdsm/lib/vdsm/network/netlink/link.py", line 56, in get_link 18:56:15 with _get_link(name=name, sock=sock) as link: 18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 18:56:15 return next(self.gen) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_ 64/vdsm/lib/vdsm/network/netlink/link.py", line 199, in _get_link 18:56:15 err = _rtnl_link_get_kernel(sock, index, name, byref(link)) 18:56:15 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type 18:56:15 -------------------- >> begin captured logging <<
18:56:15 2016-09-20 18:55:55,220 WARNING [py.warnings] (MainThread) /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead 18:56:15 inspect.getargspec(func) 18:56:15 (inspect:1041) 18:56:15 2016-09-20 18:55:55,222 INFO [root] (MainThread) Bond check_pkopm has been created. (bond:142) 18:56:15 2016-09-20 18:55:55,233 INFO [root] (MainThread) Bond check_pkopm has been destroyed. (bond:149) 18:56:15 2016-09-20 18:55:55,234 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_gmayN type dummy (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_p3yZR type dummy (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,250 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 2016-09-20 18:55:55,258 INFO [root] (MainThread) Bond bond_Siiakr has been created. (bond:142) 18:56:15 2016-09-20 18:55:55,265 INFO [root] (MainThread) Bond bond_Siiakr has been destroyed. (bond:149) 18:56:15 2016-09-20 18:55:55,265 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_gmayN (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,277 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 2016-09-20 18:55:55,278 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_p3yZR (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,289 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 --------------------- >> end captured logging <<
I think we should get rid of the ctypes code, it is very hard to understand, unsafe, and slow. If we need to call C we should use proper python extensions. On Thu, Sep 22, 2016 at 5:34 PM, Piotr Kliczewski <pkliczew@redhat.com> wrote:
On Thu, Sep 22, 2016 at 9:58 AM, Dan Kenigsberg <danken@redhat.com> wrote:
On Wed, Sep 21, 2016 at 12:18:37AM +0300, Nir Soffer wrote:
On Tue, Sep 20, 2016 at 8:18 PM, Nir Soffer <nsoffer@redhat.com> wrote:
Failed only one patch when building 4, so this looks like a random failure, but the error is disturbing.
Petr, could you look into it? Could it be that libnl3 changed their API? Or that a slave is pulling an ancient lib? Or broken kernel? Nothing makes sense.
See http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x8
6_64/2255/console
17:00:31 ============================================================
==========
17:00:31 ERROR: test_bond_create_failure_on_slave_add (network.link_bond_test.LinkBondTests) 17:00:31 ------------------------------------------------------------
17:00:31 Traceback (most recent call last): 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/tests/network/link_bond_test.py", line 83, in test_bond_create_failure_on_slave_add 17:00:31 base_bond.add_slaves((nic1, nic2)) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/bond.py", line 153, in add_slaves 17:00:31 with _preserve_iface_state(slave): 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 17:00:31 return next(self.gen) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/bond.py", line 214, in _preserve_iface_state 17:00:31 dev_was_up = iface.is_up(dev) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/iface.py", line 54, in is_up 17:00:31 return is_admin_up(dev) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/iface.py", line 58, in is_admin_up 17:00:31 return is_link_up(get_link(dev)['flags'], check_oper_status=False) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/netlink/link.py", line 56, in get_link 17:00:31 with _get_link(name=name, sock=sock) as link: 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 17:00:31 return next(self.gen) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/netlink/link.py", line 199, in _get_link 17:00:31 err = _rtnl_link_get_kernel(sock, index, name, byref(link)) 17:00:31 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type 17:00:31 -------------------- >> begin captured logging <<
17:00:31 2016-09-20 17:00:12,586 WARNING [py.warnings] (MainThread) /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead 17:00:31 inspect.getargspec(func) 17:00:31 (inspect:1041)
Piotr, the DeprecationWarning is unrelated to our code but we are using this API, too.
Please take a look at https://gerrit.ovirt.org/64313
17:00:31 2016-09-20 17:00:12,587 INFO [root] (MainThread) Bond check_7lzvb has been created. (bond:142) 17:00:31 2016-09-20 17:00:12,598 INFO [root] (MainThread) Bond check_7lzvb has been destroyed. (bond:149) 17:00:31 2016-09-20 17:00:12,598 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_Dmwc5 type dummy (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,607 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,608 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_xIK5C type dummy (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,615 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,617 INFO [root] (MainThread) Bond bond_rwTTFu has been created. (bond:142) 17:00:31 2016-09-20 17:00:12,634 INFO [root] (MainThread) Bond bond_rwTTFu has been destroyed. (bond:149) 17:00:31 2016-09-20 17:00:12,634 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_Dmwc5 (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,658 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,659 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_xIK5C (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,676 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 --------------------- >> end captured logging <<
Another instance, again only one patch failed when building 5 patches
series.
This time with new log format for the tests.
18:56:15 ============================================================
==========
18:56:15 ERROR: test_bond_create_failure_on_slave_add (network.link_bond_test.LinkBondTests) 18:56:15 ------------------------------------------------------------
18:56:15 Traceback (most recent call last): 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/tests/network/link_bond_test.py", line 83, in test_bond_create_failure_on_slave_add 18:56:15 base_bond.add_slaves((nic1, nic2)) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/bond.py", line 153, in add_slaves 18:56:15 with _preserve_iface_state(slave): 18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 18:56:15 return next(self.gen) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/bond.py", line 214, in _preserve_iface_state 18:56:15 dev_was_up = iface.is_up(dev) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/iface.py", line 54, in is_up 18:56:15 return is_admin_up(dev) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/iface.py", line 58, in is_admin_up 18:56:15 return is_link_up(get_link(dev)['flags'], check_oper_status=False) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/netlink/link.py", line 56, in get_link 18:56:15 with _get_link(name=name, sock=sock) as link: 18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 18:56:15 return next(self.gen) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/netlink/link.py", line 199, in _get_link 18:56:15 err = _rtnl_link_get_kernel(sock, index, name, byref(link)) 18:56:15 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type 18:56:15 -------------------- >> begin captured logging <<
18:56:15 2016-09-20 18:55:55,220 WARNING [py.warnings] (MainThread) /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead 18:56:15 inspect.getargspec(func) 18:56:15 (inspect:1041) 18:56:15 2016-09-20 18:55:55,222 INFO [root] (MainThread) Bond check_pkopm has been created. (bond:142) 18:56:15 2016-09-20 18:55:55,233 INFO [root] (MainThread) Bond check_pkopm has been destroyed. (bond:149) 18:56:15 2016-09-20 18:55:55,234 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_gmayN type dummy (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_p3yZR type dummy (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,250 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 2016-09-20 18:55:55,258 INFO [root] (MainThread) Bond bond_Siiakr has been created. (bond:142) 18:56:15 2016-09-20 18:55:55,265 INFO [root] (MainThread) Bond bond_Siiakr has been destroyed. (bond:149) 18:56:15 2016-09-20 18:55:55,265 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_gmayN (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,277 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 2016-09-20 18:55:55,278 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_p3yZR (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,289 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 --------------------- >> end captured logging <<
On Thu, Sep 22, 2016 at 05:38:48PM +0300, Nir Soffer wrote:
I think we should get rid of the ctypes code, it is very hard to understand, unsafe, and slow.
If we need to call C we should use proper python extensions.
Correct. https://trello.com/c/HtmyG8kk/74-use-python-libnl3-instead-of-our-own-ctype-... added to our trello. Tomas, does python-libnl3 have bindings to handle netlink events? I'm looking for an equivalent to our netlink monitor https://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=lib/vdsm/network/netlink...
On Thu, Sep 22, 2016 at 5:34 PM, Piotr Kliczewski <pkliczew@redhat.com> wrote:
On Thu, Sep 22, 2016 at 9:58 AM, Dan Kenigsberg <danken@redhat.com> wrote:
On Wed, Sep 21, 2016 at 12:18:37AM +0300, Nir Soffer wrote:
On Tue, Sep 20, 2016 at 8:18 PM, Nir Soffer <nsoffer@redhat.com> wrote:
Failed only one patch when building 4, so this looks like a random failure, but the error is disturbing.
Petr, could you look into it? Could it be that libnl3 changed their API? Or that a slave is pulling an ancient lib? Or broken kernel? Nothing makes sense.
See http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x8
6_64/2255/console
17:00:31 ============================================================
==========
17:00:31 ERROR: test_bond_create_failure_on_slave_add (network.link_bond_test.LinkBondTests) 17:00:31 ------------------------------------------------------------
17:00:31 Traceback (most recent call last): 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/tests/network/link_bond_test.py", line 83, in test_bond_create_failure_on_slave_add 17:00:31 base_bond.add_slaves((nic1, nic2)) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/bond.py", line 153, in add_slaves 17:00:31 with _preserve_iface_state(slave): 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 17:00:31 return next(self.gen) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/bond.py", line 214, in _preserve_iface_state 17:00:31 dev_was_up = iface.is_up(dev) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/iface.py", line 54, in is_up 17:00:31 return is_admin_up(dev) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/iface.py", line 58, in is_admin_up 17:00:31 return is_link_up(get_link(dev)['flags'], check_oper_status=False) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/netlink/link.py", line 56, in get_link 17:00:31 with _get_link(name=name, sock=sock) as link: 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 17:00:31 return next(self.gen) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/netlink/link.py", line 199, in _get_link 17:00:31 err = _rtnl_link_get_kernel(sock, index, name, byref(link)) 17:00:31 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type 17:00:31 -------------------- >> begin captured logging <<
17:00:31 2016-09-20 17:00:12,586 WARNING [py.warnings] (MainThread) /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead 17:00:31 inspect.getargspec(func) 17:00:31 (inspect:1041)
Piotr, the DeprecationWarning is unrelated to our code but we are using this API, too.
Please take a look at https://gerrit.ovirt.org/64313
17:00:31 2016-09-20 17:00:12,587 INFO [root] (MainThread) Bond check_7lzvb has been created. (bond:142) 17:00:31 2016-09-20 17:00:12,598 INFO [root] (MainThread) Bond check_7lzvb has been destroyed. (bond:149) 17:00:31 2016-09-20 17:00:12,598 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_Dmwc5 type dummy (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,607 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,608 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_xIK5C type dummy (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,615 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,617 INFO [root] (MainThread) Bond bond_rwTTFu has been created. (bond:142) 17:00:31 2016-09-20 17:00:12,634 INFO [root] (MainThread) Bond bond_rwTTFu has been destroyed. (bond:149) 17:00:31 2016-09-20 17:00:12,634 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_Dmwc5 (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,658 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,659 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_xIK5C (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,676 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 --------------------- >> end captured logging <<
Another instance, again only one patch failed when building 5 patches
series.
This time with new log format for the tests.
18:56:15 ============================================================
==========
18:56:15 ERROR: test_bond_create_failure_on_slave_add (network.link_bond_test.LinkBondTests) 18:56:15 ------------------------------------------------------------
18:56:15 Traceback (most recent call last): 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/tests/network/link_bond_test.py", line 83, in test_bond_create_failure_on_slave_add 18:56:15 base_bond.add_slaves((nic1, nic2)) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/bond.py", line 153, in add_slaves 18:56:15 with _preserve_iface_state(slave): 18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 18:56:15 return next(self.gen) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/bond.py", line 214, in _preserve_iface_state 18:56:15 dev_was_up = iface.is_up(dev) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/iface.py", line 54, in is_up 18:56:15 return is_admin_up(dev) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/iface.py", line 58, in is_admin_up 18:56:15 return is_link_up(get_link(dev)['flags'], check_oper_status=False) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/netlink/link.py", line 56, in get_link 18:56:15 with _get_link(name=name, sock=sock) as link: 18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 18:56:15 return next(self.gen) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/netlink/link.py", line 199, in _get_link 18:56:15 err = _rtnl_link_get_kernel(sock, index, name, byref(link)) 18:56:15 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type 18:56:15 -------------------- >> begin captured logging <<
18:56:15 2016-09-20 18:55:55,220 WARNING [py.warnings] (MainThread) /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead 18:56:15 inspect.getargspec(func) 18:56:15 (inspect:1041) 18:56:15 2016-09-20 18:55:55,222 INFO [root] (MainThread) Bond check_pkopm has been created. (bond:142) 18:56:15 2016-09-20 18:55:55,233 INFO [root] (MainThread) Bond check_pkopm has been destroyed. (bond:149) 18:56:15 2016-09-20 18:55:55,234 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_gmayN type dummy (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_p3yZR type dummy (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,250 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 2016-09-20 18:55:55,258 INFO [root] (MainThread) Bond bond_Siiakr has been created. (bond:142) 18:56:15 2016-09-20 18:55:55,265 INFO [root] (MainThread) Bond bond_Siiakr has been destroyed. (bond:149) 18:56:15 2016-09-20 18:55:55,265 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_gmayN (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,277 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 2016-09-20 18:55:55,278 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_p3yZR (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,289 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 --------------------- >> end captured logging <<
--=-67cwno0plYCf4UiDPw0o Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, 2016-09-22 at 18:14 +0300, Dan Kenigsberg wrote:
On Thu, Sep 22, 2016 at 05:38:48PM +0300, Nir Soffer wrote:
=20 I think we should get rid of the ctypes code, it is very hard to understand, unsafe, and slow. =20 If we need to call C we should use proper python extensions. =20 Correct. https://trello.com/c/HtmyG8kk/74-use-python-libnl3-instead-of-our-own -ctype-based-code added to our trello. =20 Tomas, does python-libnl3 have bindings to handle netlink events? I'm looking for an equivalent to our netlink monitor https://gerrit.ovirt.org/gitweb?p=3Dvdsm.git;a=3Dblob;f=3Dlib/vdsm/networ= k/ netlink/monitor.py;h=3D380e5c93d513467e82f6a345fb976d75a61a36bc;hb=3DHEAD
=20 =20
=20 =20 On Thu, Sep 22, 2016 at 5:34 PM, Piotr Kliczewski <pkliczew@redhat. com> wrote: =20
=20 =20 =20 On Thu, Sep 22, 2016 at 9:58 AM, Dan Kenigsberg <danken@redhat.co m> wrote: =20
=20 On Tue, Sep 20, 2016 at 8:18 PM, Nir Soffer <nsoffer@redhat.c om> wrote:
=20 Failed only one patch when building 4, so this looks like a random failure, but the error is disturbing. =20 Petr, could you look into it? Could it be that libnl3 changed
=20 On Wed, Sep 21, 2016 at 12:18:37AM +0300, Nir Soffer wrote: their API? Or that a slave is pulling an ancient lib? Or broken kernel? Nothing makes sense. =20
=20
=20 =20 See http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc 24-x8 6_64/2255/console =20 =20 =20 17:00:31 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 =20 17:00:31 ERROR: test_bond_create_failure_on_slave_add (network.link_bond_test.LinkBondTests) 17:00:31 -------------------------------------------------- ----------
=20
=20 17:00:31 Traceback (most recent call last): 17:00:31=C2=A0=C2=A0=C2=A0File "/home/jenkins/workspace/vdsm_master_check-patch-fc24- x86_64 /vdsm/tests/network/link_bond_test.py", =20 =20 line 83, in test_bond_create_failure_on_slave_add 17:00:31=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0base_bond.add_slaves((nic= 1, nic2)) 17:00:31=C2=A0=C2=A0=C2=A0File "/home/jenkins/workspace/vdsm_master_check-patch-fc24- x86_64 /vdsm/lib/vdsm/network/link/bond.py", =20 =20 line 153, in add_slaves 17:00:31=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0with _preserve_iface_stat= e(slave): 17:00:31=C2=A0=C2=A0=C2=A0File "/usr/lib64/python3.5/contextlib= .py", line 59, in __enter__ =20 =20 17:00:31=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0return next(self.gen) 17:00:31=C2=A0=C2=A0=C2=A0File "/home/jenkins/workspace/vdsm_master_check-patch-fc24- x86_64 /vdsm/lib/vdsm/network/link/bond.py", =20 =20 line 214, in _preserve_iface_state 17:00:31=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0dev_was_up =3D iface.is_u=
Hi Dan, the libnl3 package is somewhat experimental. someone added it, and eventually I got around to package it for Fedora. I don't know who is using it and how well it works. There are no bug-reports about anything not working, but maybe just nobody is using it. I don't know. I strongly suspect if you want to use it, you'd have to improve it. On the other hand, I'd be open to merge stuff upstream and bring it to RHEL. For rhel-7.3 I just rebased libnl3 package. I think we could get the python-package to rhel-7.4 if we improve it sufficiently (if you plan to do that, we should put that on the agenda early enough). You certainly know Python better then me, so have a look... Does that answer the question? Thomas p(dev)
17:00:31=C2=A0=C2=A0=C2=A0File "/home/jenkins/workspace/vdsm_master_check-patch-fc24- x86_64 /vdsm/lib/vdsm/network/link/iface.py", =20 =20 line 54, in is_up 17:00:31=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0return is_admin_up(dev) 17:00:31=C2=A0=C2=A0=C2=A0File "/home/jenkins/workspace/vdsm_master_check-patch-fc24- x86_64 /vdsm/lib/vdsm/network/link/iface.py", =20 =20 line 58, in is_admin_up 17:00:31=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0return is_link_up(get_lin= k(dev)['flags'], check_oper_status=3DFalse) =20 =20 17:00:31=C2=A0=C2=A0=C2=A0File "/home/jenkins/workspace/vdsm_master_check-patch-fc24- x86_64 /vdsm/lib/vdsm/network/netlink/link.py", =20 =20 line 56, in get_link 17:00:31=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0with _get_link(name=3Dnam= e, sock=3Dsock) as link: 17:00:31=C2=A0=C2=A0=C2=A0File "/usr/lib64/python3.5/contextlib= .py", line 59, in __enter__ =20 =20 17:00:31=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0return next(self.gen) 17:00:31=C2=A0=C2=A0=C2=A0File "/home/jenkins/workspace/vdsm_master_check-patch-fc24- x86_64 /vdsm/lib/vdsm/network/netlink/link.py", =20 =20 line 199, in _get_link 17:00:31=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0err =3D _rtnl_link_get_ke= rnel(sock, index, name, byref(link)) =20 =20 17:00:31 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type =20 =20 17:00:31 -------------------- >> begin captured logging <<
=20 17:00:31 2016-09-20 17:00:12,586 WARNING [py.warnings] (MainThread) /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning: =20 =20 inspect.getargspec() is deprecated, use inspect.signature() instead 17:00:31=C2=A0=C2=A0=C2=A0inspect.getargspec(func) 17:00:31=C2=A0=C2=A0(inspect:1041) =20 Piotr, the DeprecationWarning is unrelated to our code but we are using
=20 this API, too. =20 =20 Please take a look at https://gerrit.ovirt.org/64313 =20 =20 =20 =20
=20 17:00:31 2016-09-20 17:00:12,587 INFO=C2=A0=C2=A0[root] (MainTh= read) Bond check_7lzvb has been created. (bond:142) 17:00:31 2016-09-20 17:00:12,598 INFO=C2=A0=C2=A0[root] (MainTh= read) Bond check_7lzvb has been destroyed. (bond:149) 17:00:31 2016-09-20 17:00:12,598 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_Dmwc5 type dummy (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,607 DEBUG [root] (MainThread) SUCCESS: <err> =3D b''; <rc> =3D 0 (commands:93) 17:00:31 2016-09-20 17:00:12,608 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_xIK5C type dummy (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,615 DEBUG [root] (MainThread) SUCCESS: <err> =3D b''; <rc> =3D 0 (commands:93) 17:00:31 2016-09-20 17:00:12,617 INFO=C2=A0=C2=A0[root] (MainTh= read) Bond bond_rwTTFu has been created. (bond:142) 17:00:31 2016-09-20 17:00:12,634 INFO=C2=A0=C2=A0[root] (MainTh= read) Bond bond_rwTTFu has been destroyed. (bond:149) 17:00:31 2016-09-20 17:00:12,634 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_Dmwc5 (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,658 DEBUG [root] (MainThread) SUCCESS: <err> =3D b''; <rc> =3D 0 (commands:93) 17:00:31 2016-09-20 17:00:12,659 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_xIK5C (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,676 DEBUG [root] (MainThread) SUCCESS: <err> =3D b''; <rc> =3D 0 (commands:93) 17:00:31 --------------------- >> end captured logging <<
=20 =20 Another instance, again only one patch failed when building 5 patches series. =20 =20 This time with new log format for the tests. =20 18:56:15 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 18:56:15 ERROR: test_bond_create_failure_on_slave_add (network.link_bond_test.LinkBondTests) 18:56:15 ---------------------------------------------------- --------
=20 18:56:15 Traceback (most recent call last): 18:56:15=C2=A0=C2=A0=C2=A0File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/tests/network/link_bond_test.py", =20 line 83, in test_bond_create_failure_on_slave_add 18:56:15=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0base_bond.add_slaves((nic1,= nic2)) 18:56:15=C2=A0=C2=A0=C2=A0File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/bond.py", =20 line 153, in add_slaves 18:56:15=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0with _preserve_iface_state(= slave): 18:56:15=C2=A0=C2=A0=C2=A0File "/usr/lib64/python3.5/contextlib.p= y", line 59, in __enter__ =20 18:56:15=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0return next(self.gen) 18:56:15=C2=A0=C2=A0=C2=A0File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/bond.py", =20 line 214, in _preserve_iface_state 18:56:15=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0dev_was_up =3D iface.is_up(= dev) 18:56:15=C2=A0=C2=A0=C2=A0File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/iface.py", =20 line 54, in is_up 18:56:15=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0return is_admin_up(dev) 18:56:15=C2=A0=C2=A0=C2=A0File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/iface.py", =20 line 58, in is_admin_up 18:56:15=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0return is_link_up(get_link(= dev)['flags'], check_oper_status=3DFalse) =20 18:56:15=C2=A0=C2=A0=C2=A0File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/netlink/link.py", =20 line 56, in get_link 18:56:15=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0with _get_link(name=3Dname,= sock=3Dsock) as link: 18:56:15=C2=A0=C2=A0=C2=A0File "/usr/lib64/python3.5/contextlib.p= y", line 59, in __enter__ =20 18:56:15=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0return next(self.gen) 18:56:15=C2=A0=C2=A0=C2=A0File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/netlink/link.py", =20 line 199, in _get_link 18:56:15=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0err =3D _rtnl_link_get_kern= el(sock, index, name, byref(link)) 18:56:15 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type =20 18:56:15 -------------------- >> begin captured logging <<
=20 18:56:15 2016-09-20 18:55:55,220 WARNING [py.warnings] (MainThread) /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead 18:56:15=C2=A0=C2=A0=C2=A0inspect.getargspec(func) 18:56:15=C2=A0=C2=A0(inspect:1041) 18:56:15 2016-09-20 18:55:55,222 INFO=C2=A0=C2=A0[root] (MainThre= ad) Bond check_pkopm has been created. (bond:142) 18:56:15 2016-09-20 18:55:55,233 INFO=C2=A0=C2=A0[root] (MainThre= ad) Bond check_pkopm has been destroyed. (bond:149) 18:56:15 2016-09-20 18:55:55,234 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_gmayN type dummy (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) SUCCESS: <err> =3D b''; <rc> =3D 0 (commands:93) 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_p3yZR type dummy (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,250 DEBUG [root] (MainThread) SUCCESS: <err> =3D b''; <rc> =3D 0 (commands:93) 18:56:15 2016-09-20 18:55:55,258 INFO=C2=A0=C2=A0[root] (MainThre= ad) Bond bond_Siiakr has been created. (bond:142) 18:56:15 2016-09-20 18:55:55,265 INFO=C2=A0=C2=A0[root] (MainThre= ad) Bond bond_Siiakr has been destroyed. (bond:149) 18:56:15 2016-09-20 18:55:55,265 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_gmayN (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,277 DEBUG [root] (MainThread) SUCCESS: <err> =3D b''; <rc> =3D 0 (commands:93) 18:56:15 2016-09-20 18:55:55,278 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_p3yZR (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,289 DEBUG [root] (MainThread) SUCCESS: <err> =3D b''; <rc> =3D 0 (commands:93) 18:56:15 --------------------- >> end captured logging <<
=20 =20 =20 --=-67cwno0plYCf4UiDPw0o Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJX4/geAAoJECnCNm5N/FcoWxkP/j8ivxektJKmnfO3Ca5VCtgP gBzE1P5URuDgHEtdrom3HV/NT9fGFVosB4xZmPOjmtqgLmhwrf7cBr/ZqWu2DV5P 0RCbKILImo31puX3qECHXPCbv9yo5W9cW0qsnIbHDCqxNGy+UukC0YsfCjMZ2OGO rKVN3zmrVxHZHoizmO9OZ8AjzeKPxz64UFHiAlavljp4Zkj8wv/rzT7d5g40vqCM aqrTVgASBvt16rvbOdWU3ZY4rmUsiE8cByRShsjJ84NMlaumW3CG0ohvEXVrg/MQ G+TDFUL6YF0Coicw2kw/v3Wu5y73T6rVDPTCcUOBkQ6AZOA3VjekbcNsTT5JU7+z pL3u1L2k4DMomUKzTHiBkFoik2eaV3fwLtsCZXzW3VR+ddSfztB2fn5UR68hQsik pnNmLr7dERmQPu/BZr8KGkpJTdI51DVNaq62u8/kxm1TsXd2/Bo9kkkvSCLEqtgQ a2zeHkoE1Yh3BPnPM+bg0kYPfNn2aL3hVB/KbuQBkA8ivEZ4jL5hxUh8t+/2Ci4/ y99JpIVd6Z3S3tym003MVGjXNc0cMkItxk/faThy9PpyQD+0RnwCccMJKWW3bkGV O6Ve8Qnidzh9nUY4AUpba6lbL4N7V0Ha/9nL7lSy8vHkOpaisLCtjWVUxysmIK+1 svplWXnhrVLFLZbIs29N =X39r -----END PGP SIGNATURE----- --=-67cwno0plYCf4UiDPw0o--
And here: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2337/consol... On Thu, Sep 22, 2016 at 5:34 PM, Piotr Kliczewski <pkliczew@redhat.com> wrote:
On Thu, Sep 22, 2016 at 9:58 AM, Dan Kenigsberg <danken@redhat.com> wrote:
On Wed, Sep 21, 2016 at 12:18:37AM +0300, Nir Soffer wrote:
On Tue, Sep 20, 2016 at 8:18 PM, Nir Soffer <nsoffer@redhat.com> wrote:
Failed only one patch when building 4, so this looks like a random failure, but the error is disturbing.
Petr, could you look into it? Could it be that libnl3 changed their API? Or that a slave is pulling an ancient lib? Or broken kernel? Nothing makes sense.
See http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x8
6_64/2255/console
17:00:31 ============================================================
==========
17:00:31 ERROR: test_bond_create_failure_on_slave_add (network.link_bond_test.LinkBondTests) 17:00:31 ------------------------------------------------------------
17:00:31 Traceback (most recent call last): 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/tests/network/link_bond_test.py", line 83, in test_bond_create_failure_on_slave_add 17:00:31 base_bond.add_slaves((nic1, nic2)) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/bond.py", line 153, in add_slaves 17:00:31 with _preserve_iface_state(slave): 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 17:00:31 return next(self.gen) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/bond.py", line 214, in _preserve_iface_state 17:00:31 dev_was_up = iface.is_up(dev) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/iface.py", line 54, in is_up 17:00:31 return is_admin_up(dev) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/iface.py", line 58, in is_admin_up 17:00:31 return is_link_up(get_link(dev)['flags'], check_oper_status=False) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/netlink/link.py", line 56, in get_link 17:00:31 with _get_link(name=name, sock=sock) as link: 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 17:00:31 return next(self.gen) 17:00:31 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/netlink/link.py", line 199, in _get_link 17:00:31 err = _rtnl_link_get_kernel(sock, index, name, byref(link)) 17:00:31 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type 17:00:31 -------------------- >> begin captured logging <<
17:00:31 2016-09-20 17:00:12,586 WARNING [py.warnings] (MainThread) /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead 17:00:31 inspect.getargspec(func) 17:00:31 (inspect:1041)
Piotr, the DeprecationWarning is unrelated to our code but we are using this API, too.
Please take a look at https://gerrit.ovirt.org/64313
17:00:31 2016-09-20 17:00:12,587 INFO [root] (MainThread) Bond check_7lzvb has been created. (bond:142) 17:00:31 2016-09-20 17:00:12,598 INFO [root] (MainThread) Bond check_7lzvb has been destroyed. (bond:149) 17:00:31 2016-09-20 17:00:12,598 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_Dmwc5 type dummy (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,607 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,608 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_xIK5C type dummy (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,615 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,617 INFO [root] (MainThread) Bond bond_rwTTFu has been created. (bond:142) 17:00:31 2016-09-20 17:00:12,634 INFO [root] (MainThread) Bond bond_rwTTFu has been destroyed. (bond:149) 17:00:31 2016-09-20 17:00:12,634 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_Dmwc5 (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,658 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 2016-09-20 17:00:12,659 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_xIK5C (cwd None) (commands:69) 17:00:31 2016-09-20 17:00:12,676 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 17:00:31 --------------------- >> end captured logging <<
Another instance, again only one patch failed when building 5 patches
series.
This time with new log format for the tests.
18:56:15 ============================================================
==========
18:56:15 ERROR: test_bond_create_failure_on_slave_add (network.link_bond_test.LinkBondTests) 18:56:15 ------------------------------------------------------------
18:56:15 Traceback (most recent call last): 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/tests/network/link_bond_test.py", line 83, in test_bond_create_failure_on_slave_add 18:56:15 base_bond.add_slaves((nic1, nic2)) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/bond.py", line 153, in add_slaves 18:56:15 with _preserve_iface_state(slave): 18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 18:56:15 return next(self.gen) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/bond.py", line 214, in _preserve_iface_state 18:56:15 dev_was_up = iface.is_up(dev) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/iface.py", line 54, in is_up 18:56:15 return is_admin_up(dev) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/link/iface.py", line 58, in is_admin_up 18:56:15 return is_link_up(get_link(dev)['flags'], check_oper_status=False) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/netlink/link.py", line 56, in get_link 18:56:15 with _get_link(name=name, sock=sock) as link: 18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__ 18:56:15 return next(self.gen) 18:56:15 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 /vdsm/lib/vdsm/network/netlink/link.py", line 199, in _get_link 18:56:15 err = _rtnl_link_get_kernel(sock, index, name, byref(link)) 18:56:15 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type 18:56:15 -------------------- >> begin captured logging <<
18:56:15 2016-09-20 18:55:55,220 WARNING [py.warnings] (MainThread) /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead 18:56:15 inspect.getargspec(func) 18:56:15 (inspect:1041) 18:56:15 2016-09-20 18:55:55,222 INFO [root] (MainThread) Bond check_pkopm has been created. (bond:142) 18:56:15 2016-09-20 18:55:55,233 INFO [root] (MainThread) Bond check_pkopm has been destroyed. (bond:149) 18:56:15 2016-09-20 18:55:55,234 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_gmayN type dummy (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_p3yZR type dummy (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,250 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 2016-09-20 18:55:55,258 INFO [root] (MainThread) Bond bond_Siiakr has been created. (bond:142) 18:56:15 2016-09-20 18:55:55,265 INFO [root] (MainThread) Bond bond_Siiakr has been destroyed. (bond:149) 18:56:15 2016-09-20 18:55:55,265 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_gmayN (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,277 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 2016-09-20 18:55:55,278 DEBUG [root] (MainThread) /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_p3yZR (cwd None) (commands:69) 18:56:15 2016-09-20 18:55:55,289 DEBUG [root] (MainThread) SUCCESS: <err> = b''; <rc> = 0 (commands:93) 18:56:15 --------------------- >> end captured logging <<
participants (4)
-
Dan Kenigsberg -
Nir Soffer -
Piotr Kliczewski -
Thomas Haller