2:58 a.m.
On Wed, Sep 21, 2016 at 12:18:37AM +0300, Nir Soffer wrote:
On Tue, Sep 20, 2016 at 8:18 PM, Nir Soffer
<nsoffer(a)redhat.com> wrote:
> Failed only one patch when building 4, so this looks like
> a random failure, but the error is disturbing.
Petr, could you look into it? Could it be that libnl3 changed their API?
Or that a slave is pulling an ancient lib? Or broken kernel? Nothing
makes sense.
>
> See http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2255/con...
>
> 17:00:31 ======================================================================
> 17:00:31 ERROR: test_bond_create_failure_on_slave_add
> (network.link_bond_test.LinkBondTests)
> 17:00:31 ----------------------------------------------------------------------
> 17:00:31 Traceback (most recent call last):
> 17:00:31 File
>
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/tests/network/link_bond_test.py",
> line 83, in test_bond_create_failure_on_slave_add
> 17:00:31 base_bond.add_slaves((nic1, nic2))
> 17:00:31 File
>
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py",
> line 153, in add_slaves
> 17:00:31 with _preserve_iface_state(slave):
> 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in
__enter__
> 17:00:31 return next(self.gen)
> 17:00:31 File
>
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py",
> line 214, in _preserve_iface_state
> 17:00:31 dev_was_up = iface.is_up(dev)
> 17:00:31 File
>
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py",
> line 54, in is_up
> 17:00:31 return is_admin_up(dev)
> 17:00:31 File
>
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py",
> line 58, in is_admin_up
> 17:00:31 return is_link_up(get_link(dev)['flags'],
check_oper_status=False)
> 17:00:31 File
>
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py",
> line 56, in get_link
> 17:00:31 with _get_link(name=name, sock=sock) as link:
> 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in
__enter__
> 17:00:31 return next(self.gen)
> 17:00:31 File
>
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py",
> line 199, in _get_link
> 17:00:31 err = _rtnl_link_get_kernel(sock, index, name, byref(link))
> 17:00:31 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong
type
> 17:00:31 -------------------- >> begin captured logging <<
--------------------
> 17:00:31 2016-09-20 17:00:12,586 WARNING [py.warnings] (MainThread)
> /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning:
> inspect.getargspec() is deprecated, use inspect.signature() instead
> 17:00:31 inspect.getargspec(func)
> 17:00:31 (inspect:1041)
Piotr, the DeprecationWarning is unrelated to our code but we are using
this API, too.
> 17:00:31 2016-09-20 17:00:12,587 INFO [root] (MainThread) Bond
> check_7lzvb has been created. (bond:142)
> 17:00:31 2016-09-20 17:00:12,598 INFO [root] (MainThread) Bond
> check_7lzvb has been destroyed. (bond:149)
> 17:00:31 2016-09-20 17:00:12,598 DEBUG [root] (MainThread)
> /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_Dmwc5
> type dummy (cwd None) (commands:69)
> 17:00:31 2016-09-20 17:00:12,607 DEBUG [root] (MainThread) SUCCESS:
> <err> = b''; <rc> = 0 (commands:93)
> 17:00:31 2016-09-20 17:00:12,608 DEBUG [root] (MainThread)
> /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_xIK5C
> type dummy (cwd None) (commands:69)
> 17:00:31 2016-09-20 17:00:12,615 DEBUG [root] (MainThread) SUCCESS:
> <err> = b''; <rc> = 0 (commands:93)
> 17:00:31 2016-09-20 17:00:12,617 INFO [root] (MainThread) Bond
> bond_rwTTFu has been created. (bond:142)
> 17:00:31 2016-09-20 17:00:12,634 INFO [root] (MainThread) Bond
> bond_rwTTFu has been destroyed. (bond:149)
> 17:00:31 2016-09-20 17:00:12,634 DEBUG [root] (MainThread)
> /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_Dmwc5
> (cwd None) (commands:69)
> 17:00:31 2016-09-20 17:00:12,658 DEBUG [root] (MainThread) SUCCESS:
> <err> = b''; <rc> = 0 (commands:93)
> 17:00:31 2016-09-20 17:00:12,659 DEBUG [root] (MainThread)
> /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_xIK5C
> (cwd None) (commands:69)
> 17:00:31 2016-09-20 17:00:12,676 DEBUG [root] (MainThread) SUCCESS:
> <err> = b''; <rc> = 0 (commands:93)
> 17:00:31 --------------------- >> end captured logging <<
---------------------
Another instance, again only one patch failed when building 5 patches series.
This time with new log format for the tests.
18:56:15 ======================================================================
18:56:15 ERROR: test_bond_create_failure_on_slave_add
(network.link_bond_test.LinkBondTests)
18:56:15 ----------------------------------------------------------------------
18:56:15 Traceback (most recent call last):
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/tests/network/link_bond_test.py",
line 83, in test_bond_create_failure_on_slave_add
18:56:15 base_bond.add_slaves((nic1, nic2))
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py",
line 153, in add_slaves
18:56:15 with _preserve_iface_state(slave):
18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__
18:56:15 return next(self.gen)
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py",
line 214, in _preserve_iface_state
18:56:15 dev_was_up = iface.is_up(dev)
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py",
line 54, in is_up
18:56:15 return is_admin_up(dev)
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py",
line 58, in is_admin_up
18:56:15 return is_link_up(get_link(dev)['flags'], check_oper_status=False)
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py",
line 56, in get_link
18:56:15 with _get_link(name=name, sock=sock) as link:
18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__
18:56:15 return next(self.gen)
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py",
line 199, in _get_link
18:56:15 err = _rtnl_link_get_kernel(sock, index, name, byref(link))
18:56:15 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type
18:56:15 -------------------- >> begin captured logging <<
--------------------
18:56:15 2016-09-20 18:55:55,220 WARNING [py.warnings] (MainThread)
/usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning:
inspect.getargspec() is deprecated, use inspect.signature() instead
18:56:15 inspect.getargspec(func)
18:56:15 (inspect:1041)
18:56:15 2016-09-20 18:55:55,222 INFO [root] (MainThread) Bond
check_pkopm has been created. (bond:142)
18:56:15 2016-09-20 18:55:55,233 INFO [root] (MainThread) Bond
check_pkopm has been destroyed. (bond:149)
18:56:15 2016-09-20 18:55:55,234 DEBUG [root] (MainThread)
/usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_gmayN
type dummy (cwd None) (commands:69)
18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) SUCCESS:
<err> = b''; <rc> = 0 (commands:93)
18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread)
/usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_p3yZR
type dummy (cwd None) (commands:69)
18:56:15 2016-09-20 18:55:55,250 DEBUG [root] (MainThread) SUCCESS:
<err> = b''; <rc> = 0 (commands:93)
18:56:15 2016-09-20 18:55:55,258 INFO [root] (MainThread) Bond
bond_Siiakr has been created. (bond:142)
18:56:15 2016-09-20 18:55:55,265 INFO [root] (MainThread) Bond
bond_Siiakr has been destroyed. (bond:149)
18:56:15 2016-09-20 18:55:55,265 DEBUG [root] (MainThread)
/usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_gmayN (cwd
None) (commands:69)
18:56:15 2016-09-20 18:55:55,277 DEBUG [root] (MainThread) SUCCESS:
<err> = b''; <rc> = 0 (commands:93)
18:56:15 2016-09-20 18:55:55,278 DEBUG [root] (MainThread)
/usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_p3yZR (cwd
None) (commands:69)
18:56:15 2016-09-20 18:55:55,289 DEBUG [root] (MainThread) SUCCESS:
<err> = b''; <rc> = 0 (commands:93)
18:56:15 --------------------- >> end captured logging <<
---------------------