1:18 p.m.
Failed only one patch when building 4, so this looks like
a random failure, but the error is disturbing.
See http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2255/con...
17:00:31 ======================================================================
17:00:31 ERROR: test_bond_create_failure_on_slave_add
(network.link_bond_test.LinkBondTests)
17:00:31 ----------------------------------------------------------------------
17:00:31 Traceback (most recent call last):
17:00:31 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/tests/network/link_bond_test.py",
line 83, in test_bond_create_failure_on_slave_add
17:00:31 base_bond.add_slaves((nic1, nic2))
17:00:31 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py",
line 153, in add_slaves
17:00:31 with _preserve_iface_state(slave):
17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__
17:00:31 return next(self.gen)
17:00:31 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py",
line 214, in _preserve_iface_state
17:00:31 dev_was_up = iface.is_up(dev)
17:00:31 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py",
line 54, in is_up
17:00:31 return is_admin_up(dev)
17:00:31 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py",
line 58, in is_admin_up
17:00:31 return is_link_up(get_link(dev)['flags'], check_oper_status=False)
17:00:31 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py",
line 56, in get_link
17:00:31 with _get_link(name=name, sock=sock) as link:
17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__
17:00:31 return next(self.gen)
17:00:31 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py",
line 199, in _get_link
17:00:31 err = _rtnl_link_get_kernel(sock, index, name, byref(link))
17:00:31 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type
17:00:31 -------------------- >> begin captured logging <<
--------------------
17:00:31 2016-09-20 17:00:12,586 WARNING [py.warnings] (MainThread)
/usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning:
inspect.getargspec() is deprecated, use inspect.signature() instead
17:00:31 inspect.getargspec(func)
17:00:31 (inspect:1041)
17:00:31 2016-09-20 17:00:12,587 INFO [root] (MainThread) Bond
check_7lzvb has been created. (bond:142)
17:00:31 2016-09-20 17:00:12,598 INFO [root] (MainThread) Bond
check_7lzvb has been destroyed. (bond:149)
17:00:31 2016-09-20 17:00:12,598 DEBUG [root] (MainThread)
/usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_Dmwc5
type dummy (cwd None) (commands:69)
17:00:31 2016-09-20 17:00:12,607 DEBUG [root] (MainThread) SUCCESS:
<err> = b''; <rc> = 0 (commands:93)
17:00:31 2016-09-20 17:00:12,608 DEBUG [root] (MainThread)
/usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_xIK5C
type dummy (cwd None) (commands:69)
17:00:31 2016-09-20 17:00:12,615 DEBUG [root] (MainThread) SUCCESS:
<err> = b''; <rc> = 0 (commands:93)
17:00:31 2016-09-20 17:00:12,617 INFO [root] (MainThread) Bond
bond_rwTTFu has been created. (bond:142)
17:00:31 2016-09-20 17:00:12,634 INFO [root] (MainThread) Bond
bond_rwTTFu has been destroyed. (bond:149)
17:00:31 2016-09-20 17:00:12,634 DEBUG [root] (MainThread)
/usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_Dmwc5
(cwd None) (commands:69)
17:00:31 2016-09-20 17:00:12,658 DEBUG [root] (MainThread) SUCCESS:
<err> = b''; <rc> = 0 (commands:93)
17:00:31 2016-09-20 17:00:12,659 DEBUG [root] (MainThread)
/usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_xIK5C
(cwd None) (commands:69)
17:00:31 2016-09-20 17:00:12,676 DEBUG [root] (MainThread) SUCCESS:
<err> = b''; <rc> = 0 (commands:93)
17:00:31 --------------------- >> end captured logging <<
---------------------
5:18 p.m.
On Tue, Sep 20, 2016 at 8:18 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
Failed only one patch when building 4, so this looks like
a random failure, but the error is disturbing.
See http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2255/con...
17:00:31 ======================================================================
17:00:31 ERROR: test_bond_create_failure_on_slave_add
(network.link_bond_test.LinkBondTests)
17:00:31 ----------------------------------------------------------------------
17:00:31 Traceback (most recent call last):
17:00:31 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/tests/network/link_bond_test.py",
line 83, in test_bond_create_failure_on_slave_add
17:00:31 base_bond.add_slaves((nic1, nic2))
17:00:31 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py",
line 153, in add_slaves
17:00:31 with _preserve_iface_state(slave):
17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__
17:00:31 return next(self.gen)
17:00:31 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py",
line 214, in _preserve_iface_state
17:00:31 dev_was_up = iface.is_up(dev)
17:00:31 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py",
line 54, in is_up
17:00:31 return is_admin_up(dev)
17:00:31 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py",
line 58, in is_admin_up
17:00:31 return is_link_up(get_link(dev)['flags'], check_oper_status=False)
17:00:31 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py",
line 56, in get_link
17:00:31 with _get_link(name=name, sock=sock) as link:
17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__
17:00:31 return next(self.gen)
17:00:31 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py",
line 199, in _get_link
17:00:31 err = _rtnl_link_get_kernel(sock, index, name, byref(link))
17:00:31 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type
17:00:31 -------------------- >> begin captured logging <<
--------------------
17:00:31 2016-09-20 17:00:12,586 WARNING [py.warnings] (MainThread)
/usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning:
inspect.getargspec() is deprecated, use inspect.signature() instead
17:00:31 inspect.getargspec(func)
17:00:31 (inspect:1041)
17:00:31 2016-09-20 17:00:12,587 INFO [root] (MainThread) Bond
check_7lzvb has been created. (bond:142)
17:00:31 2016-09-20 17:00:12,598 INFO [root] (MainThread) Bond
check_7lzvb has been destroyed. (bond:149)
17:00:31 2016-09-20 17:00:12,598 DEBUG [root] (MainThread)
/usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_Dmwc5
type dummy (cwd None) (commands:69)
17:00:31 2016-09-20 17:00:12,607 DEBUG [root] (MainThread) SUCCESS:
<err> = b''; <rc> = 0 (commands:93)
17:00:31 2016-09-20 17:00:12,608 DEBUG [root] (MainThread)
/usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_xIK5C
type dummy (cwd None) (commands:69)
17:00:31 2016-09-20 17:00:12,615 DEBUG [root] (MainThread) SUCCESS:
<err> = b''; <rc> = 0 (commands:93)
17:00:31 2016-09-20 17:00:12,617 INFO [root] (MainThread) Bond
bond_rwTTFu has been created. (bond:142)
17:00:31 2016-09-20 17:00:12,634 INFO [root] (MainThread) Bond
bond_rwTTFu has been destroyed. (bond:149)
17:00:31 2016-09-20 17:00:12,634 DEBUG [root] (MainThread)
/usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_Dmwc5
(cwd None) (commands:69)
17:00:31 2016-09-20 17:00:12,658 DEBUG [root] (MainThread) SUCCESS:
<err> = b''; <rc> = 0 (commands:93)
17:00:31 2016-09-20 17:00:12,659 DEBUG [root] (MainThread)
/usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_xIK5C
(cwd None) (commands:69)
17:00:31 2016-09-20 17:00:12,676 DEBUG [root] (MainThread) SUCCESS:
<err> = b''; <rc> = 0 (commands:93)
17:00:31 --------------------- >> end captured logging <<
---------------------
Another instance, again only one patch failed when building 5 patches series.
This time with new log format for the tests.
18:56:15 ======================================================================
18:56:15 ERROR: test_bond_create_failure_on_slave_add
(network.link_bond_test.LinkBondTests)
18:56:15 ----------------------------------------------------------------------
18:56:15 Traceback (most recent call last):
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/tests/network/link_bond_test.py",
line 83, in test_bond_create_failure_on_slave_add
18:56:15 base_bond.add_slaves((nic1, nic2))
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py",
line 153, in add_slaves
18:56:15 with _preserve_iface_state(slave):
18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__
18:56:15 return next(self.gen)
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py",
line 214, in _preserve_iface_state
18:56:15 dev_was_up = iface.is_up(dev)
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py",
line 54, in is_up
18:56:15 return is_admin_up(dev)
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py",
line 58, in is_admin_up
18:56:15 return is_link_up(get_link(dev)['flags'], check_oper_status=False)
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py",
line 56, in get_link
18:56:15 with _get_link(name=name, sock=sock) as link:
18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__
18:56:15 return next(self.gen)
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py",
line 199, in _get_link
18:56:15 err = _rtnl_link_get_kernel(sock, index, name, byref(link))
18:56:15 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type
18:56:15 -------------------- >> begin captured logging <<
--------------------
18:56:15 2016-09-20 18:55:55,220 WARNING [py.warnings] (MainThread)
/usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning:
inspect.getargspec() is deprecated, use inspect.signature() instead
18:56:15 inspect.getargspec(func)
18:56:15 (inspect:1041)
18:56:15 2016-09-20 18:55:55,222 INFO [root] (MainThread) Bond
check_pkopm has been created. (bond:142)
18:56:15 2016-09-20 18:55:55,233 INFO [root] (MainThread) Bond
check_pkopm has been destroyed. (bond:149)
18:56:15 2016-09-20 18:55:55,234 DEBUG [root] (MainThread)
/usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_gmayN
type dummy (cwd None) (commands:69)
18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) SUCCESS:
<err> = b''; <rc> = 0 (commands:93)
18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread)
/usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_p3yZR
type dummy (cwd None) (commands:69)
18:56:15 2016-09-20 18:55:55,250 DEBUG [root] (MainThread) SUCCESS:
<err> = b''; <rc> = 0 (commands:93)
18:56:15 2016-09-20 18:55:55,258 INFO [root] (MainThread) Bond
bond_Siiakr has been created. (bond:142)
18:56:15 2016-09-20 18:55:55,265 INFO [root] (MainThread) Bond
bond_Siiakr has been destroyed. (bond:149)
18:56:15 2016-09-20 18:55:55,265 DEBUG [root] (MainThread)
/usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_gmayN (cwd
None) (commands:69)
18:56:15 2016-09-20 18:55:55,277 DEBUG [root] (MainThread) SUCCESS:
<err> = b''; <rc> = 0 (commands:93)
18:56:15 2016-09-20 18:55:55,278 DEBUG [root] (MainThread)
/usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_p3yZR (cwd
None) (commands:69)
18:56:15 2016-09-20 18:55:55,289 DEBUG [root] (MainThread) SUCCESS:
<err> = b''; <rc> = 0 (commands:93)
18:56:15 --------------------- >> end captured logging <<
---------------------
3:58 a.m.
On Wed, Sep 21, 2016 at 12:18:37AM +0300, Nir Soffer wrote:
On Tue, Sep 20, 2016 at 8:18 PM, Nir Soffer
<nsoffer(a)redhat.com> wrote:
> Failed only one patch when building 4, so this looks like
> a random failure, but the error is disturbing.
Petr, could you look into it? Could it be that libnl3 changed their API?
Or that a slave is pulling an ancient lib? Or broken kernel? Nothing
makes sense.
>
> See http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2255/con...
>
> 17:00:31 ======================================================================
> 17:00:31 ERROR: test_bond_create_failure_on_slave_add
> (network.link_bond_test.LinkBondTests)
> 17:00:31 ----------------------------------------------------------------------
> 17:00:31 Traceback (most recent call last):
> 17:00:31 File
>
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/tests/network/link_bond_test.py",
> line 83, in test_bond_create_failure_on_slave_add
> 17:00:31 base_bond.add_slaves((nic1, nic2))
> 17:00:31 File
>
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py",
> line 153, in add_slaves
> 17:00:31 with _preserve_iface_state(slave):
> 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in
__enter__
> 17:00:31 return next(self.gen)
> 17:00:31 File
>
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py",
> line 214, in _preserve_iface_state
> 17:00:31 dev_was_up = iface.is_up(dev)
> 17:00:31 File
>
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py",
> line 54, in is_up
> 17:00:31 return is_admin_up(dev)
> 17:00:31 File
>
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py",
> line 58, in is_admin_up
> 17:00:31 return is_link_up(get_link(dev)['flags'],
check_oper_status=False)
> 17:00:31 File
>
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py",
> line 56, in get_link
> 17:00:31 with _get_link(name=name, sock=sock) as link:
> 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in
__enter__
> 17:00:31 return next(self.gen)
> 17:00:31 File
>
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py",
> line 199, in _get_link
> 17:00:31 err = _rtnl_link_get_kernel(sock, index, name, byref(link))
> 17:00:31 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong
type
> 17:00:31 -------------------- >> begin captured logging <<
--------------------
> 17:00:31 2016-09-20 17:00:12,586 WARNING [py.warnings] (MainThread)
> /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning:
> inspect.getargspec() is deprecated, use inspect.signature() instead
> 17:00:31 inspect.getargspec(func)
> 17:00:31 (inspect:1041)
Piotr, the DeprecationWarning is unrelated to our code but we are using
this API, too.
> 17:00:31 2016-09-20 17:00:12,587 INFO [root] (MainThread) Bond
> check_7lzvb has been created. (bond:142)
> 17:00:31 2016-09-20 17:00:12,598 INFO [root] (MainThread) Bond
> check_7lzvb has been destroyed. (bond:149)
> 17:00:31 2016-09-20 17:00:12,598 DEBUG [root] (MainThread)
> /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_Dmwc5
> type dummy (cwd None) (commands:69)
> 17:00:31 2016-09-20 17:00:12,607 DEBUG [root] (MainThread) SUCCESS:
> <err> = b''; <rc> = 0 (commands:93)
> 17:00:31 2016-09-20 17:00:12,608 DEBUG [root] (MainThread)
> /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_xIK5C
> type dummy (cwd None) (commands:69)
> 17:00:31 2016-09-20 17:00:12,615 DEBUG [root] (MainThread) SUCCESS:
> <err> = b''; <rc> = 0 (commands:93)
> 17:00:31 2016-09-20 17:00:12,617 INFO [root] (MainThread) Bond
> bond_rwTTFu has been created. (bond:142)
> 17:00:31 2016-09-20 17:00:12,634 INFO [root] (MainThread) Bond
> bond_rwTTFu has been destroyed. (bond:149)
> 17:00:31 2016-09-20 17:00:12,634 DEBUG [root] (MainThread)
> /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_Dmwc5
> (cwd None) (commands:69)
> 17:00:31 2016-09-20 17:00:12,658 DEBUG [root] (MainThread) SUCCESS:
> <err> = b''; <rc> = 0 (commands:93)
> 17:00:31 2016-09-20 17:00:12,659 DEBUG [root] (MainThread)
> /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_xIK5C
> (cwd None) (commands:69)
> 17:00:31 2016-09-20 17:00:12,676 DEBUG [root] (MainThread) SUCCESS:
> <err> = b''; <rc> = 0 (commands:93)
> 17:00:31 --------------------- >> end captured logging <<
---------------------
Another instance, again only one patch failed when building 5 patches series.
This time with new log format for the tests.
18:56:15 ======================================================================
18:56:15 ERROR: test_bond_create_failure_on_slave_add
(network.link_bond_test.LinkBondTests)
18:56:15 ----------------------------------------------------------------------
18:56:15 Traceback (most recent call last):
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/tests/network/link_bond_test.py",
line 83, in test_bond_create_failure_on_slave_add
18:56:15 base_bond.add_slaves((nic1, nic2))
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py",
line 153, in add_slaves
18:56:15 with _preserve_iface_state(slave):
18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__
18:56:15 return next(self.gen)
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/bond.py",
line 214, in _preserve_iface_state
18:56:15 dev_was_up = iface.is_up(dev)
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py",
line 54, in is_up
18:56:15 return is_admin_up(dev)
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/link/iface.py",
line 58, in is_admin_up
18:56:15 return is_link_up(get_link(dev)['flags'], check_oper_status=False)
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py",
line 56, in get_link
18:56:15 with _get_link(name=name, sock=sock) as link:
18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__
18:56:15 return next(self.gen)
18:56:15 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib/vdsm/network/netlink/link.py",
line 199, in _get_link
18:56:15 err = _rtnl_link_get_kernel(sock, index, name, byref(link))
18:56:15 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type
18:56:15 -------------------- >> begin captured logging <<
--------------------
18:56:15 2016-09-20 18:55:55,220 WARNING [py.warnings] (MainThread)
/usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning:
inspect.getargspec() is deprecated, use inspect.signature() instead
18:56:15 inspect.getargspec(func)
18:56:15 (inspect:1041)
18:56:15 2016-09-20 18:55:55,222 INFO [root] (MainThread) Bond
check_pkopm has been created. (bond:142)
18:56:15 2016-09-20 18:55:55,233 INFO [root] (MainThread) Bond
check_pkopm has been destroyed. (bond:149)
18:56:15 2016-09-20 18:55:55,234 DEBUG [root] (MainThread)
/usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_gmayN
type dummy (cwd None) (commands:69)
18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) SUCCESS:
<err> = b''; <rc> = 0 (commands:93)
18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread)
/usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_p3yZR
type dummy (cwd None) (commands:69)
18:56:15 2016-09-20 18:55:55,250 DEBUG [root] (MainThread) SUCCESS:
<err> = b''; <rc> = 0 (commands:93)
18:56:15 2016-09-20 18:55:55,258 INFO [root] (MainThread) Bond
bond_Siiakr has been created. (bond:142)
18:56:15 2016-09-20 18:55:55,265 INFO [root] (MainThread) Bond
bond_Siiakr has been destroyed. (bond:149)
18:56:15 2016-09-20 18:55:55,265 DEBUG [root] (MainThread)
/usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_gmayN (cwd
None) (commands:69)
18:56:15 2016-09-20 18:55:55,277 DEBUG [root] (MainThread) SUCCESS:
<err> = b''; <rc> = 0 (commands:93)
18:56:15 2016-09-20 18:55:55,278 DEBUG [root] (MainThread)
/usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_p3yZR (cwd
None) (commands:69)
18:56:15 2016-09-20 18:55:55,289 DEBUG [root] (MainThread) SUCCESS:
<err> = b''; <rc> = 0 (commands:93)
18:56:15 --------------------- >> end captured logging <<
---------------------
10:34 a.m.
On Thu, Sep 22, 2016 at 9:58 AM, Dan Kenigsberg <danken(a)redhat.com> wrote:
On Wed, Sep 21, 2016 at 12:18:37AM +0300, Nir Soffer wrote:
> On Tue, Sep 20, 2016 at 8:18 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
> > Failed only one patch when building 4, so this looks like
> > a random failure, but the error is disturbing.
Petr, could you look into it? Could it be that libnl3 changed their API?
Or that a slave is pulling an ancient lib? Or broken kernel? Nothing
makes sense.
> >
> > See http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-
x86_64/2255/console
> >
> > 17:00:31 ============================================================
==========
> > 17:00:31 ERROR: test_bond_create_failure_on_slave_add
> > (network.link_bond_test.LinkBondTests)
> > 17:00:31 ------------------------------------------------------------
----------
> > 17:00:31 Traceback (most recent call last):
> > 17:00:31 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_
64/vdsm/tests/network/link_bond_test.py",
> > line 83, in test_bond_create_failure_on_slave_add
> > 17:00:31 base_bond.add_slaves((nic1, nic2))
> > 17:00:31 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_
64/vdsm/lib/vdsm/network/link/bond.py",
> > line 153, in add_slaves
> > 17:00:31 with _preserve_iface_state(slave):
> > 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in
__enter__
> > 17:00:31 return next(self.gen)
> > 17:00:31 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_
64/vdsm/lib/vdsm/network/link/bond.py",
> > line 214, in _preserve_iface_state
> > 17:00:31 dev_was_up = iface.is_up(dev)
> > 17:00:31 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_
64/vdsm/lib/vdsm/network/link/iface.py",
> > line 54, in is_up
> > 17:00:31 return is_admin_up(dev)
> > 17:00:31 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_
64/vdsm/lib/vdsm/network/link/iface.py",
> > line 58, in is_admin_up
> > 17:00:31 return is_link_up(get_link(dev)['flags'],
check_oper_status=False)
> > 17:00:31 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_
64/vdsm/lib/vdsm/network/netlink/link.py",
> > line 56, in get_link
> > 17:00:31 with _get_link(name=name, sock=sock) as link:
> > 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59, in
__enter__
> > 17:00:31 return next(self.gen)
> > 17:00:31 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_
64/vdsm/lib/vdsm/network/netlink/link.py",
> > line 199, in _get_link
> > 17:00:31 err = _rtnl_link_get_kernel(sock, index, name,
byref(link))
> > 17:00:31 ctypes.ArgumentError: argument 3: <class 'TypeError'>:
wrong
type
> > 17:00:31 -------------------- >> begin captured logging <<
--------------------
> > 17:00:31 2016-09-20 17:00:12,586 WARNING [py.warnings] (MainThread)
> > /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning:
> > inspect.getargspec() is deprecated, use inspect.signature() instead
> > 17:00:31 inspect.getargspec(func)
> > 17:00:31 (inspect:1041)
Piotr, the DeprecationWarning is unrelated to our code but we are using
this API, too.
Please take a look at https://gerrit.ovirt.org/64313
> > 17:00:31 2016-09-20 17:00:12,587 INFO [root] (MainThread)
Bond
> > check_7lzvb has been created. (bond:142)
> > 17:00:31 2016-09-20 17:00:12,598 INFO [root] (MainThread) Bond
> > check_7lzvb has been destroyed. (bond:149)
> > 17:00:31 2016-09-20 17:00:12,598 DEBUG [root] (MainThread)
> > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_Dmwc5
> > type dummy (cwd None) (commands:69)
> > 17:00:31 2016-09-20 17:00:12,607 DEBUG [root] (MainThread) SUCCESS:
> > <err> = b''; <rc> = 0 (commands:93)
> > 17:00:31 2016-09-20 17:00:12,608 DEBUG [root] (MainThread)
> > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_xIK5C
> > type dummy (cwd None) (commands:69)
> > 17:00:31 2016-09-20 17:00:12,615 DEBUG [root] (MainThread) SUCCESS:
> > <err> = b''; <rc> = 0 (commands:93)
> > 17:00:31 2016-09-20 17:00:12,617 INFO [root] (MainThread) Bond
> > bond_rwTTFu has been created. (bond:142)
> > 17:00:31 2016-09-20 17:00:12,634 INFO [root] (MainThread) Bond
> > bond_rwTTFu has been destroyed. (bond:149)
> > 17:00:31 2016-09-20 17:00:12,634 DEBUG [root] (MainThread)
> > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_Dmwc5
> > (cwd None) (commands:69)
> > 17:00:31 2016-09-20 17:00:12,658 DEBUG [root] (MainThread) SUCCESS:
> > <err> = b''; <rc> = 0 (commands:93)
> > 17:00:31 2016-09-20 17:00:12,659 DEBUG [root] (MainThread)
> > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_xIK5C
> > (cwd None) (commands:69)
> > 17:00:31 2016-09-20 17:00:12,676 DEBUG [root] (MainThread) SUCCESS:
> > <err> = b''; <rc> = 0 (commands:93)
> > 17:00:31 --------------------- >> end captured logging <<
---------------------
>
> Another instance, again only one patch failed when building 5 patches
series.
>
> This time with new log format for the tests.
>
> 18:56:15 ============================================================
==========
> 18:56:15 ERROR: test_bond_create_failure_on_slave_add
> (network.link_bond_test.LinkBondTests)
> 18:56:15 ------------------------------------------------------------
----------
> 18:56:15 Traceback (most recent call last):
> 18:56:15 File
> "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_
64/vdsm/tests/network/link_bond_test.py",
> line 83, in test_bond_create_failure_on_slave_add
> 18:56:15 base_bond.add_slaves((nic1, nic2))
> 18:56:15 File
> "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_
64/vdsm/lib/vdsm/network/link/bond.py",
> line 153, in add_slaves
> 18:56:15 with _preserve_iface_state(slave):
> 18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in
__enter__
> 18:56:15 return next(self.gen)
> 18:56:15 File
> "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_
64/vdsm/lib/vdsm/network/link/bond.py",
> line 214, in _preserve_iface_state
> 18:56:15 dev_was_up = iface.is_up(dev)
> 18:56:15 File
> "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_
64/vdsm/lib/vdsm/network/link/iface.py",
> line 54, in is_up
> 18:56:15 return is_admin_up(dev)
> 18:56:15 File
> "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_
64/vdsm/lib/vdsm/network/link/iface.py",
> line 58, in is_admin_up
> 18:56:15 return is_link_up(get_link(dev)['flags'],
check_oper_status=False)
> 18:56:15 File
> "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_
64/vdsm/lib/vdsm/network/netlink/link.py",
> line 56, in get_link
> 18:56:15 with _get_link(name=name, sock=sock) as link:
> 18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in
__enter__
> 18:56:15 return next(self.gen)
> 18:56:15 File
> "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_
64/vdsm/lib/vdsm/network/netlink/link.py",
> line 199, in _get_link
> 18:56:15 err = _rtnl_link_get_kernel(sock, index, name, byref(link))
> 18:56:15 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong
type
> 18:56:15 -------------------- >> begin captured logging <<
--------------------
> 18:56:15 2016-09-20 18:55:55,220 WARNING [py.warnings] (MainThread)
> /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning:
> inspect.getargspec() is deprecated, use inspect.signature() instead
> 18:56:15 inspect.getargspec(func)
> 18:56:15 (inspect:1041)
> 18:56:15 2016-09-20 18:55:55,222 INFO [root] (MainThread) Bond
> check_pkopm has been created. (bond:142)
> 18:56:15 2016-09-20 18:55:55,233 INFO [root] (MainThread) Bond
> check_pkopm has been destroyed. (bond:149)
> 18:56:15 2016-09-20 18:55:55,234 DEBUG [root] (MainThread)
> /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_gmayN
> type dummy (cwd None) (commands:69)
> 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) SUCCESS:
> <err> = b''; <rc> = 0 (commands:93)
> 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread)
> /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_p3yZR
> type dummy (cwd None) (commands:69)
> 18:56:15 2016-09-20 18:55:55,250 DEBUG [root] (MainThread) SUCCESS:
> <err> = b''; <rc> = 0 (commands:93)
> 18:56:15 2016-09-20 18:55:55,258 INFO [root] (MainThread) Bond
> bond_Siiakr has been created. (bond:142)
> 18:56:15 2016-09-20 18:55:55,265 INFO [root] (MainThread) Bond
> bond_Siiakr has been destroyed. (bond:149)
> 18:56:15 2016-09-20 18:55:55,265 DEBUG [root] (MainThread)
> /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_gmayN (cwd
> None) (commands:69)
> 18:56:15 2016-09-20 18:55:55,277 DEBUG [root] (MainThread) SUCCESS:
> <err> = b''; <rc> = 0 (commands:93)
> 18:56:15 2016-09-20 18:55:55,278 DEBUG [root] (MainThread)
> /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_p3yZR (cwd
> None) (commands:69)
> 18:56:15 2016-09-20 18:55:55,289 DEBUG [root] (MainThread) SUCCESS:
> <err> = b''; <rc> = 0 (commands:93)
> 18:56:15 --------------------- >> end captured logging <<
---------------------
10:38 a.m.
I think we should get rid of the ctypes code, it is very hard to
understand, unsafe, and slow.
If we need to call C we should use proper python extensions.
On Thu, Sep 22, 2016 at 5:34 PM, Piotr Kliczewski <pkliczew(a)redhat.com>
wrote:
On Thu, Sep 22, 2016 at 9:58 AM, Dan Kenigsberg <danken(a)redhat.com> wrote:
> On Wed, Sep 21, 2016 at 12:18:37AM +0300, Nir Soffer wrote:
> > On Tue, Sep 20, 2016 at 8:18 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
> > > Failed only one patch when building 4, so this looks like
> > > a random failure, but the error is disturbing.
>
> Petr, could you look into it? Could it be that libnl3 changed their API?
> Or that a slave is pulling an ancient lib? Or broken kernel? Nothing
> makes sense.
>
> > >
> > > See http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x8
> 6_64/2255/console
> > >
> > > 17:00:31 ============================================================
> ==========
> > > 17:00:31 ERROR: test_bond_create_failure_on_slave_add
> > > (network.link_bond_test.LinkBondTests)
> > > 17:00:31 ------------------------------------------------------------
> ----------
> > > 17:00:31 Traceback (most recent call last):
> > > 17:00:31 File
> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/tests/network/link_bond_test.py",
> > > line 83, in test_bond_create_failure_on_slave_add
> > > 17:00:31 base_bond.add_slaves((nic1, nic2))
> > > 17:00:31 File
> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/link/bond.py",
> > > line 153, in add_slaves
> > > 17:00:31 with _preserve_iface_state(slave):
> > > 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59,
in
> __enter__
> > > 17:00:31 return next(self.gen)
> > > 17:00:31 File
> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/link/bond.py",
> > > line 214, in _preserve_iface_state
> > > 17:00:31 dev_was_up = iface.is_up(dev)
> > > 17:00:31 File
> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/link/iface.py",
> > > line 54, in is_up
> > > 17:00:31 return is_admin_up(dev)
> > > 17:00:31 File
> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/link/iface.py",
> > > line 58, in is_admin_up
> > > 17:00:31 return is_link_up(get_link(dev)['flags'],
> check_oper_status=False)
> > > 17:00:31 File
> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/netlink/link.py",
> > > line 56, in get_link
> > > 17:00:31 with _get_link(name=name, sock=sock) as link:
> > > 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59,
in
> __enter__
> > > 17:00:31 return next(self.gen)
> > > 17:00:31 File
> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/netlink/link.py",
> > > line 199, in _get_link
> > > 17:00:31 err = _rtnl_link_get_kernel(sock, index, name,
> byref(link))
> > > 17:00:31 ctypes.ArgumentError: argument 3: <class
'TypeError'>: wrong
> type
> > > 17:00:31 -------------------- >> begin captured logging <<
> --------------------
> > > 17:00:31 2016-09-20 17:00:12,586 WARNING [py.warnings] (MainThread)
> > > /usr/lib/python3.5/site-packages/nose/util.py:453:
> DeprecationWarning:
> > > inspect.getargspec() is deprecated, use inspect.signature() instead
> > > 17:00:31 inspect.getargspec(func)
> > > 17:00:31 (inspect:1041)
>
> Piotr, the DeprecationWarning is unrelated to our code but we are using
> this API, too.
>
>
Please take a look at https://gerrit.ovirt.org/64313
> > > 17:00:31 2016-09-20 17:00:12,587 INFO [root] (MainThread) Bond
> > > check_7lzvb has been created. (bond:142)
> > > 17:00:31 2016-09-20 17:00:12,598 INFO [root] (MainThread) Bond
> > > check_7lzvb has been destroyed. (bond:149)
> > > 17:00:31 2016-09-20 17:00:12,598 DEBUG [root] (MainThread)
> > > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_Dmwc5
> > > type dummy (cwd None) (commands:69)
> > > 17:00:31 2016-09-20 17:00:12,607 DEBUG [root] (MainThread) SUCCESS:
> > > <err> = b''; <rc> = 0 (commands:93)
> > > 17:00:31 2016-09-20 17:00:12,608 DEBUG [root] (MainThread)
> > > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_xIK5C
> > > type dummy (cwd None) (commands:69)
> > > 17:00:31 2016-09-20 17:00:12,615 DEBUG [root] (MainThread) SUCCESS:
> > > <err> = b''; <rc> = 0 (commands:93)
> > > 17:00:31 2016-09-20 17:00:12,617 INFO [root] (MainThread) Bond
> > > bond_rwTTFu has been created. (bond:142)
> > > 17:00:31 2016-09-20 17:00:12,634 INFO [root] (MainThread) Bond
> > > bond_rwTTFu has been destroyed. (bond:149)
> > > 17:00:31 2016-09-20 17:00:12,634 DEBUG [root] (MainThread)
> > > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_Dmwc5
> > > (cwd None) (commands:69)
> > > 17:00:31 2016-09-20 17:00:12,658 DEBUG [root] (MainThread) SUCCESS:
> > > <err> = b''; <rc> = 0 (commands:93)
> > > 17:00:31 2016-09-20 17:00:12,659 DEBUG [root] (MainThread)
> > > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_xIK5C
> > > (cwd None) (commands:69)
> > > 17:00:31 2016-09-20 17:00:12,676 DEBUG [root] (MainThread) SUCCESS:
> > > <err> = b''; <rc> = 0 (commands:93)
> > > 17:00:31 --------------------- >> end captured logging <<
> ---------------------
> >
> > Another instance, again only one patch failed when building 5 patches
> series.
> >
> > This time with new log format for the tests.
> >
> > 18:56:15 ============================================================
> ==========
> > 18:56:15 ERROR: test_bond_create_failure_on_slave_add
> > (network.link_bond_test.LinkBondTests)
> > 18:56:15 ------------------------------------------------------------
> ----------
> > 18:56:15 Traceback (most recent call last):
> > 18:56:15 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/tests/network/link_bond_test.py",
> > line 83, in test_bond_create_failure_on_slave_add
> > 18:56:15 base_bond.add_slaves((nic1, nic2))
> > 18:56:15 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/link/bond.py",
> > line 153, in add_slaves
> > 18:56:15 with _preserve_iface_state(slave):
> > 18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in
> __enter__
> > 18:56:15 return next(self.gen)
> > 18:56:15 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/link/bond.py",
> > line 214, in _preserve_iface_state
> > 18:56:15 dev_was_up = iface.is_up(dev)
> > 18:56:15 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/link/iface.py",
> > line 54, in is_up
> > 18:56:15 return is_admin_up(dev)
> > 18:56:15 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/link/iface.py",
> > line 58, in is_admin_up
> > 18:56:15 return is_link_up(get_link(dev)['flags'],
> check_oper_status=False)
> > 18:56:15 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/netlink/link.py",
> > line 56, in get_link
> > 18:56:15 with _get_link(name=name, sock=sock) as link:
> > 18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in
> __enter__
> > 18:56:15 return next(self.gen)
> > 18:56:15 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/netlink/link.py",
> > line 199, in _get_link
> > 18:56:15 err = _rtnl_link_get_kernel(sock, index, name, byref(link))
> > 18:56:15 ctypes.ArgumentError: argument 3: <class 'TypeError'>:
wrong
> type
> > 18:56:15 -------------------- >> begin captured logging <<
> --------------------
> > 18:56:15 2016-09-20 18:55:55,220 WARNING [py.warnings] (MainThread)
> > /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning:
> > inspect.getargspec() is deprecated, use inspect.signature() instead
> > 18:56:15 inspect.getargspec(func)
> > 18:56:15 (inspect:1041)
> > 18:56:15 2016-09-20 18:55:55,222 INFO [root] (MainThread) Bond
> > check_pkopm has been created. (bond:142)
> > 18:56:15 2016-09-20 18:55:55,233 INFO [root] (MainThread) Bond
> > check_pkopm has been destroyed. (bond:149)
> > 18:56:15 2016-09-20 18:55:55,234 DEBUG [root] (MainThread)
> > /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_gmayN
> > type dummy (cwd None) (commands:69)
> > 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) SUCCESS:
> > <err> = b''; <rc> = 0 (commands:93)
> > 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread)
> > /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_p3yZR
> > type dummy (cwd None) (commands:69)
> > 18:56:15 2016-09-20 18:55:55,250 DEBUG [root] (MainThread) SUCCESS:
> > <err> = b''; <rc> = 0 (commands:93)
> > 18:56:15 2016-09-20 18:55:55,258 INFO [root] (MainThread) Bond
> > bond_Siiakr has been created. (bond:142)
> > 18:56:15 2016-09-20 18:55:55,265 INFO [root] (MainThread) Bond
> > bond_Siiakr has been destroyed. (bond:149)
> > 18:56:15 2016-09-20 18:55:55,265 DEBUG [root] (MainThread)
> > /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_gmayN (cwd
> > None) (commands:69)
> > 18:56:15 2016-09-20 18:55:55,277 DEBUG [root] (MainThread) SUCCESS:
> > <err> = b''; <rc> = 0 (commands:93)
> > 18:56:15 2016-09-20 18:55:55,278 DEBUG [root] (MainThread)
> > /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_p3yZR (cwd
> > None) (commands:69)
> > 18:56:15 2016-09-20 18:55:55,289 DEBUG [root] (MainThread) SUCCESS:
> > <err> = b''; <rc> = 0 (commands:93)
> > 18:56:15 --------------------- >> end captured logging <<
> ---------------------
>
11:14 a.m.
On Thu, Sep 22, 2016 at 05:38:48PM +0300, Nir Soffer wrote:
I think we should get rid of the ctypes code, it is very hard to
understand, unsafe, and slow.
If we need to call C we should use proper python extensions.
Correct.
https://trello.com/c/HtmyG8kk/74-use-python-libnl3-instead-of-our-own-cty...
added to our trello.
Tomas, does python-libnl3 have bindings to handle netlink events? I'm
looking for an equivalent to our netlink monitor
https://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=lib/vdsm/network/netl...
On Thu, Sep 22, 2016 at 5:34 PM, Piotr Kliczewski <pkliczew(a)redhat.com>
wrote:
>
>
> On Thu, Sep 22, 2016 at 9:58 AM, Dan Kenigsberg <danken(a)redhat.com> wrote:
>
>> On Wed, Sep 21, 2016 at 12:18:37AM +0300, Nir Soffer wrote:
>> > On Tue, Sep 20, 2016 at 8:18 PM, Nir Soffer <nsoffer(a)redhat.com>
wrote:
>> > > Failed only one patch when building 4, so this looks like
>> > > a random failure, but the error is disturbing.
>>
>> Petr, could you look into it? Could it be that libnl3 changed their API?
>> Or that a slave is pulling an ancient lib? Or broken kernel? Nothing
>> makes sense.
>>
>> > >
>> > > See http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x8
>> 6_64/2255/console
>> > >
>> > > 17:00:31 ============================================================
>> ==========
>> > > 17:00:31 ERROR: test_bond_create_failure_on_slave_add
>> > > (network.link_bond_test.LinkBondTests)
>> > > 17:00:31 ------------------------------------------------------------
>> ----------
>> > > 17:00:31 Traceback (most recent call last):
>> > > 17:00:31 File
>> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
>> /vdsm/tests/network/link_bond_test.py",
>> > > line 83, in test_bond_create_failure_on_slave_add
>> > > 17:00:31 base_bond.add_slaves((nic1, nic2))
>> > > 17:00:31 File
>> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
>> /vdsm/lib/vdsm/network/link/bond.py",
>> > > line 153, in add_slaves
>> > > 17:00:31 with _preserve_iface_state(slave):
>> > > 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line
59, in
>> __enter__
>> > > 17:00:31 return next(self.gen)
>> > > 17:00:31 File
>> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
>> /vdsm/lib/vdsm/network/link/bond.py",
>> > > line 214, in _preserve_iface_state
>> > > 17:00:31 dev_was_up = iface.is_up(dev)
>> > > 17:00:31 File
>> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
>> /vdsm/lib/vdsm/network/link/iface.py",
>> > > line 54, in is_up
>> > > 17:00:31 return is_admin_up(dev)
>> > > 17:00:31 File
>> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
>> /vdsm/lib/vdsm/network/link/iface.py",
>> > > line 58, in is_admin_up
>> > > 17:00:31 return is_link_up(get_link(dev)['flags'],
>> check_oper_status=False)
>> > > 17:00:31 File
>> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
>> /vdsm/lib/vdsm/network/netlink/link.py",
>> > > line 56, in get_link
>> > > 17:00:31 with _get_link(name=name, sock=sock) as link:
>> > > 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line
59, in
>> __enter__
>> > > 17:00:31 return next(self.gen)
>> > > 17:00:31 File
>> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
>> /vdsm/lib/vdsm/network/netlink/link.py",
>> > > line 199, in _get_link
>> > > 17:00:31 err = _rtnl_link_get_kernel(sock, index, name,
>> byref(link))
>> > > 17:00:31 ctypes.ArgumentError: argument 3: <class
'TypeError'>: wrong
>> type
>> > > 17:00:31 -------------------- >> begin captured logging
<<
>> --------------------
>> > > 17:00:31 2016-09-20 17:00:12,586 WARNING [py.warnings] (MainThread)
>> > > /usr/lib/python3.5/site-packages/nose/util.py:453:
>> DeprecationWarning:
>> > > inspect.getargspec() is deprecated, use inspect.signature() instead
>> > > 17:00:31 inspect.getargspec(func)
>> > > 17:00:31 (inspect:1041)
>>
>> Piotr, the DeprecationWarning is unrelated to our code but we are using
>> this API, too.
>>
>>
> Please take a look at https://gerrit.ovirt.org/64313
>
>
>> > > 17:00:31 2016-09-20 17:00:12,587 INFO [root] (MainThread) Bond
>> > > check_7lzvb has been created. (bond:142)
>> > > 17:00:31 2016-09-20 17:00:12,598 INFO [root] (MainThread) Bond
>> > > check_7lzvb has been destroyed. (bond:149)
>> > > 17:00:31 2016-09-20 17:00:12,598 DEBUG [root] (MainThread)
>> > > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_Dmwc5
>> > > type dummy (cwd None) (commands:69)
>> > > 17:00:31 2016-09-20 17:00:12,607 DEBUG [root] (MainThread) SUCCESS:
>> > > <err> = b''; <rc> = 0 (commands:93)
>> > > 17:00:31 2016-09-20 17:00:12,608 DEBUG [root] (MainThread)
>> > > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_xIK5C
>> > > type dummy (cwd None) (commands:69)
>> > > 17:00:31 2016-09-20 17:00:12,615 DEBUG [root] (MainThread) SUCCESS:
>> > > <err> = b''; <rc> = 0 (commands:93)
>> > > 17:00:31 2016-09-20 17:00:12,617 INFO [root] (MainThread) Bond
>> > > bond_rwTTFu has been created. (bond:142)
>> > > 17:00:31 2016-09-20 17:00:12,634 INFO [root] (MainThread) Bond
>> > > bond_rwTTFu has been destroyed. (bond:149)
>> > > 17:00:31 2016-09-20 17:00:12,634 DEBUG [root] (MainThread)
>> > > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_Dmwc5
>> > > (cwd None) (commands:69)
>> > > 17:00:31 2016-09-20 17:00:12,658 DEBUG [root] (MainThread) SUCCESS:
>> > > <err> = b''; <rc> = 0 (commands:93)
>> > > 17:00:31 2016-09-20 17:00:12,659 DEBUG [root] (MainThread)
>> > > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_xIK5C
>> > > (cwd None) (commands:69)
>> > > 17:00:31 2016-09-20 17:00:12,676 DEBUG [root] (MainThread) SUCCESS:
>> > > <err> = b''; <rc> = 0 (commands:93)
>> > > 17:00:31 --------------------- >> end captured logging <<
>> ---------------------
>> >
>> > Another instance, again only one patch failed when building 5 patches
>> series.
>> >
>> > This time with new log format for the tests.
>> >
>> > 18:56:15 ============================================================
>> ==========
>> > 18:56:15 ERROR: test_bond_create_failure_on_slave_add
>> > (network.link_bond_test.LinkBondTests)
>> > 18:56:15 ------------------------------------------------------------
>> ----------
>> > 18:56:15 Traceback (most recent call last):
>> > 18:56:15 File
>> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
>> /vdsm/tests/network/link_bond_test.py",
>> > line 83, in test_bond_create_failure_on_slave_add
>> > 18:56:15 base_bond.add_slaves((nic1, nic2))
>> > 18:56:15 File
>> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
>> /vdsm/lib/vdsm/network/link/bond.py",
>> > line 153, in add_slaves
>> > 18:56:15 with _preserve_iface_state(slave):
>> > 18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59,
in
>> __enter__
>> > 18:56:15 return next(self.gen)
>> > 18:56:15 File
>> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
>> /vdsm/lib/vdsm/network/link/bond.py",
>> > line 214, in _preserve_iface_state
>> > 18:56:15 dev_was_up = iface.is_up(dev)
>> > 18:56:15 File
>> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
>> /vdsm/lib/vdsm/network/link/iface.py",
>> > line 54, in is_up
>> > 18:56:15 return is_admin_up(dev)
>> > 18:56:15 File
>> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
>> /vdsm/lib/vdsm/network/link/iface.py",
>> > line 58, in is_admin_up
>> > 18:56:15 return is_link_up(get_link(dev)['flags'],
>> check_oper_status=False)
>> > 18:56:15 File
>> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
>> /vdsm/lib/vdsm/network/netlink/link.py",
>> > line 56, in get_link
>> > 18:56:15 with _get_link(name=name, sock=sock) as link:
>> > 18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59,
in
>> __enter__
>> > 18:56:15 return next(self.gen)
>> > 18:56:15 File
>> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
>> /vdsm/lib/vdsm/network/netlink/link.py",
>> > line 199, in _get_link
>> > 18:56:15 err = _rtnl_link_get_kernel(sock, index, name, byref(link))
>> > 18:56:15 ctypes.ArgumentError: argument 3: <class
'TypeError'>: wrong
>> type
>> > 18:56:15 -------------------- >> begin captured logging <<
>> --------------------
>> > 18:56:15 2016-09-20 18:55:55,220 WARNING [py.warnings] (MainThread)
>> > /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning:
>> > inspect.getargspec() is deprecated, use inspect.signature() instead
>> > 18:56:15 inspect.getargspec(func)
>> > 18:56:15 (inspect:1041)
>> > 18:56:15 2016-09-20 18:55:55,222 INFO [root] (MainThread) Bond
>> > check_pkopm has been created. (bond:142)
>> > 18:56:15 2016-09-20 18:55:55,233 INFO [root] (MainThread) Bond
>> > check_pkopm has been destroyed. (bond:149)
>> > 18:56:15 2016-09-20 18:55:55,234 DEBUG [root] (MainThread)
>> > /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_gmayN
>> > type dummy (cwd None) (commands:69)
>> > 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) SUCCESS:
>> > <err> = b''; <rc> = 0 (commands:93)
>> > 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread)
>> > /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_p3yZR
>> > type dummy (cwd None) (commands:69)
>> > 18:56:15 2016-09-20 18:55:55,250 DEBUG [root] (MainThread) SUCCESS:
>> > <err> = b''; <rc> = 0 (commands:93)
>> > 18:56:15 2016-09-20 18:55:55,258 INFO [root] (MainThread) Bond
>> > bond_Siiakr has been created. (bond:142)
>> > 18:56:15 2016-09-20 18:55:55,265 INFO [root] (MainThread) Bond
>> > bond_Siiakr has been destroyed. (bond:149)
>> > 18:56:15 2016-09-20 18:55:55,265 DEBUG [root] (MainThread)
>> > /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_gmayN (cwd
>> > None) (commands:69)
>> > 18:56:15 2016-09-20 18:55:55,277 DEBUG [root] (MainThread) SUCCESS:
>> > <err> = b''; <rc> = 0 (commands:93)
>> > 18:56:15 2016-09-20 18:55:55,278 DEBUG [root] (MainThread)
>> > /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_p3yZR (cwd
>> > None) (commands:69)
>> > 18:56:15 2016-09-20 18:55:55,289 DEBUG [root] (MainThread) SUCCESS:
>> > <err> = b''; <rc> = 0 (commands:93)
>> > 18:56:15 --------------------- >> end captured logging <<
>> ---------------------
>>
>
>
11:26 a.m.
--=-67cwno0plYCf4UiDPw0o
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
On Thu, 2016-09-22 at 18:14 +0300, Dan Kenigsberg wrote:
On Thu, Sep 22, 2016 at 05:38:48PM +0300, Nir Soffer wrote:
>=20
> I think we should get rid of the ctypes code, it is very hard to
> understand, unsafe, and slow.
>=20
> If we need to call C we should use proper python extensions.
=20
Correct.
https://trello.com/c/HtmyG8kk/74-use-python-libnl3-instead-of-our-own
-ctype-based-code
added to our trello.
=20
Tomas, does python-libnl3 have bindings to handle netlink events? I'm
looking for an equivalent to our netlink monitor
https://gerrit.ovirt.org/gitweb?p=3Dvdsm.git;a=3Dblob;f=3Dlib/vdsm/networ=
k/
netlink/monitor.py;h=3D380e5c93d513467e82f6a345fb976d75a61a36bc;hb=3DHEAD
Hi Dan,
the libnl3 package is somewhat experimental.
someone added it, and eventually I got around to package it for Fedora.
I don't know who is using it and how well it works.
There are no bug-reports about anything not working, but maybe just
nobody is using it. I don't know.
I strongly suspect if you want to use it, you'd have to improve it.
On the other hand, I'd be open to merge stuff upstream and bring it to
RHEL. For rhel-7.3 I just rebased libnl3 package. I think we could get
the python-package to rhel-7.4 if we improve it sufficiently (if you
plan to do that, we should put that on the agenda early enough).
You certainly know Python better then me, so have a look...
Does that answer the question?
Thomas
=20
=20
>=20
>=20
> On Thu, Sep 22, 2016 at 5:34 PM, Piotr Kliczewski <pkliczew@redhat.
> com>
> wrote:
>=20
> >=20
> >=20
> >=20
> > On Thu, Sep 22, 2016 at 9:58 AM, Dan Kenigsberg <danken(a)redhat.co
> > m> wrote:
> >=20
> > >=20
> > > On Wed, Sep 21, 2016 at 12:18:37AM +0300, Nir Soffer wrote:
> > > >=20
> > > > On Tue, Sep 20, 2016 at 8:18 PM, Nir Soffer <nsoffer(a)redhat.c
> > > > om> wrote:
> > > > >=20
> > > > > Failed only one patch when building 4, so this looks like
> > > > > a random failure, but the error is disturbing.
> > >=20
> > > Petr, could you look into it? Could it be that libnl3 changed
> > > their API?
> > > Or that a slave is pulling an ancient lib? Or broken kernel?
> > > Nothing
> > > makes sense.
> > >=20
> > > >=20
> > > > >=20
> > > > >=20
> > > > > See http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc
> > > > > 24-x8
> > > 6_64/2255/console
> > > >=20
> > > > >=20
> > > > >=20
> > > > > 17:00:31
> > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > > > > =3D
> > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > > >=20
> > > > >=20
> > > > > 17:00:31 ERROR: test_bond_create_failure_on_slave_add
> > > > > (network.link_bond_test.LinkBondTests)
> > > > > 17:00:31 --------------------------------------------------
> > > > > ----------
> > > ----------
> > > >=20
> > > > >=20
> > > > > 17:00:31 Traceback (most recent call last):
> > > > > 17:00:31=C2=A0=C2=A0=C2=A0File
> > > > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-
> > > > > x86_64
> > > /vdsm/tests/network/link_bond_test.py",
> > > >=20
> > > > >=20
> > > > > line 83, in test_bond_create_failure_on_slave_add
> > > > > 17:00:31=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0base_bond.add_slaves((nic=
1, nic2))
> > > > > 17:00:31=C2=A0=C2=A0=C2=A0File
> > > > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-
> > > > > x86_64
> > > /vdsm/lib/vdsm/network/link/bond.py",
> > > >=20
> > > > >=20
> > > > > line 153, in add_slaves
> > > > > 17:00:31=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0with _preserve_iface_stat=
e(slave):
> > > > > 17:00:31=C2=A0=C2=A0=C2=A0File
"/usr/lib64/python3.5/contextlib=
.py", line
> > > > > 59, in
> > > __enter__
> > > >=20
> > > > >=20
> > > > > 17:00:31=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0return next(self.gen)
> > > > > 17:00:31=C2=A0=C2=A0=C2=A0File
> > > > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-
> > > > > x86_64
> > > /vdsm/lib/vdsm/network/link/bond.py",
> > > >=20
> > > > >=20
> > > > > line 214, in _preserve_iface_state
> > > > > 17:00:31=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0dev_was_up =3D iface.is_u=
p(dev)
> > > > > 17:00:31=C2=A0=C2=A0=C2=A0File
> > > > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-
> > > > > x86_64
> > > /vdsm/lib/vdsm/network/link/iface.py",
> > > >=20
> > > > >=20
> > > > > line 54, in is_up
> > > > > 17:00:31=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0return is_admin_up(dev)
> > > > > 17:00:31=C2=A0=C2=A0=C2=A0File
> > > > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-
> > > > > x86_64
> > > /vdsm/lib/vdsm/network/link/iface.py",
> > > >=20
> > > > >=20
> > > > > line 58, in is_admin_up
> > > > > 17:00:31=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0return is_link_up(get_lin=
k(dev)['flags'],
> > > check_oper_status=3DFalse)
> > > >=20
> > > > >=20
> > > > > 17:00:31=C2=A0=C2=A0=C2=A0File
> > > > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-
> > > > > x86_64
> > > /vdsm/lib/vdsm/network/netlink/link.py",
> > > >=20
> > > > >=20
> > > > > line 56, in get_link
> > > > > 17:00:31=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0with _get_link(name=3Dnam=
e, sock=3Dsock) as link:
> > > > > 17:00:31=C2=A0=C2=A0=C2=A0File
"/usr/lib64/python3.5/contextlib=
.py", line
> > > > > 59, in
> > > __enter__
> > > >=20
> > > > >=20
> > > > > 17:00:31=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0return next(self.gen)
> > > > > 17:00:31=C2=A0=C2=A0=C2=A0File
> > > > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-
> > > > > x86_64
> > > /vdsm/lib/vdsm/network/netlink/link.py",
> > > >=20
> > > > >=20
> > > > > line 199, in _get_link
> > > > > 17:00:31=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0err =3D _rtnl_link_get_ke=
rnel(sock, index, name,
> > > byref(link))
> > > >=20
> > > > >=20
> > > > > 17:00:31 ctypes.ArgumentError: argument 3: <class
> > > > > 'TypeError'>: wrong
> > > type
> > > >=20
> > > > >=20
> > > > > 17:00:31 -------------------- >> begin captured logging
<<
> > > --------------------
> > > >=20
> > > > >=20
> > > > > 17:00:31 2016-09-20 17:00:12,586 WARNING [py.warnings]
> > > > > (MainThread)
> > > > > /usr/lib/python3.5/site-packages/nose/util.py:453:
> > > DeprecationWarning:
> > > >=20
> > > > >=20
> > > > > inspect.getargspec() is deprecated, use inspect.signature()
> > > > > instead
> > > > > 17:00:31=C2=A0=C2=A0=C2=A0inspect.getargspec(func)
> > > > > 17:00:31=C2=A0=C2=A0(inspect:1041)
> > >=20
> > > Piotr, the DeprecationWarning is unrelated to our code but we
> > > are using
> > > this API, too.
> > >=20
> > >=20
> > Please take a look at https://gerrit.ovirt.org/64313
> >=20
> >=20
> > >=20
> > > >=20
> > > > >=20
> > > > > 17:00:31 2016-09-20 17:00:12,587 INFO=C2=A0=C2=A0[root] (MainTh=
read)
> > > > > Bond
> > > > > check_7lzvb has been created. (bond:142)
> > > > > 17:00:31 2016-09-20 17:00:12,598 INFO=C2=A0=C2=A0[root] (MainTh=
read)
> > > > > Bond
> > > > > check_7lzvb has been destroyed. (bond:149)
> > > > > 17:00:31 2016-09-20 17:00:12,598 DEBUG [root] (MainThread)
> > > > > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name
> > > > > dummy_Dmwc5
> > > > > type dummy (cwd None) (commands:69)
> > > > > 17:00:31 2016-09-20 17:00:12,607 DEBUG [root] (MainThread)
> > > > > SUCCESS:
> > > > > <err> =3D b''; <rc> =3D 0 (commands:93)
> > > > > 17:00:31 2016-09-20 17:00:12,608 DEBUG [root] (MainThread)
> > > > > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name
> > > > > dummy_xIK5C
> > > > > type dummy (cwd None) (commands:69)
> > > > > 17:00:31 2016-09-20 17:00:12,615 DEBUG [root] (MainThread)
> > > > > SUCCESS:
> > > > > <err> =3D b''; <rc> =3D 0 (commands:93)
> > > > > 17:00:31 2016-09-20 17:00:12,617 INFO=C2=A0=C2=A0[root] (MainTh=
read)
> > > > > Bond
> > > > > bond_rwTTFu has been created. (bond:142)
> > > > > 17:00:31 2016-09-20 17:00:12,634 INFO=C2=A0=C2=A0[root] (MainTh=
read)
> > > > > Bond
> > > > > bond_rwTTFu has been destroyed. (bond:149)
> > > > > 17:00:31 2016-09-20 17:00:12,634 DEBUG [root] (MainThread)
> > > > > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev
> > > > > dummy_Dmwc5
> > > > > (cwd None) (commands:69)
> > > > > 17:00:31 2016-09-20 17:00:12,658 DEBUG [root] (MainThread)
> > > > > SUCCESS:
> > > > > <err> =3D b''; <rc> =3D 0 (commands:93)
> > > > > 17:00:31 2016-09-20 17:00:12,659 DEBUG [root] (MainThread)
> > > > > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev
> > > > > dummy_xIK5C
> > > > > (cwd None) (commands:69)
> > > > > 17:00:31 2016-09-20 17:00:12,676 DEBUG [root] (MainThread)
> > > > > SUCCESS:
> > > > > <err> =3D b''; <rc> =3D 0 (commands:93)
> > > > > 17:00:31 --------------------- >> end captured logging
<<
> > > ---------------------
> > > >=20
> > > >=20
> > > > Another instance, again only one patch failed when building 5
> > > > patches
> > > series.
> > > >=20
> > > >=20
> > > > This time with new log format for the tests.
> > > >=20
> > > > 18:56:15
> > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > > >=20
> > > > 18:56:15 ERROR: test_bond_create_failure_on_slave_add
> > > > (network.link_bond_test.LinkBondTests)
> > > > 18:56:15 ----------------------------------------------------
> > > > --------
> > > ----------
> > > >=20
> > > > 18:56:15 Traceback (most recent call last):
> > > > 18:56:15=C2=A0=C2=A0=C2=A0File
> > > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> > > /vdsm/tests/network/link_bond_test.py",
> > > >=20
> > > > line 83, in test_bond_create_failure_on_slave_add
> > > > 18:56:15=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0base_bond.add_slaves((nic1,=
nic2))
> > > > 18:56:15=C2=A0=C2=A0=C2=A0File
> > > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> > > /vdsm/lib/vdsm/network/link/bond.py",
> > > >=20
> > > > line 153, in add_slaves
> > > > 18:56:15=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0with _preserve_iface_state(=
slave):
> > > > 18:56:15=C2=A0=C2=A0=C2=A0File
"/usr/lib64/python3.5/contextlib.p=
y", line
> > > > 59, in
> > > __enter__
> > > >=20
> > > > 18:56:15=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0return next(self.gen)
> > > > 18:56:15=C2=A0=C2=A0=C2=A0File
> > > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> > > /vdsm/lib/vdsm/network/link/bond.py",
> > > >=20
> > > > line 214, in _preserve_iface_state
> > > > 18:56:15=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0dev_was_up =3D iface.is_up(=
dev)
> > > > 18:56:15=C2=A0=C2=A0=C2=A0File
> > > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> > > /vdsm/lib/vdsm/network/link/iface.py",
> > > >=20
> > > > line 54, in is_up
> > > > 18:56:15=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0return is_admin_up(dev)
> > > > 18:56:15=C2=A0=C2=A0=C2=A0File
> > > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> > > /vdsm/lib/vdsm/network/link/iface.py",
> > > >=20
> > > > line 58, in is_admin_up
> > > > 18:56:15=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0return is_link_up(get_link(=
dev)['flags'],
> > > check_oper_status=3DFalse)
> > > >=20
> > > > 18:56:15=C2=A0=C2=A0=C2=A0File
> > > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> > > /vdsm/lib/vdsm/network/netlink/link.py",
> > > >=20
> > > > line 56, in get_link
> > > > 18:56:15=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0with _get_link(name=3Dname,=
sock=3Dsock) as link:
> > > > 18:56:15=C2=A0=C2=A0=C2=A0File
"/usr/lib64/python3.5/contextlib.p=
y", line
> > > > 59, in
> > > __enter__
> > > >=20
> > > > 18:56:15=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0return next(self.gen)
> > > > 18:56:15=C2=A0=C2=A0=C2=A0File
> > > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> > > /vdsm/lib/vdsm/network/netlink/link.py",
> > > >=20
> > > > line 199, in _get_link
> > > > 18:56:15=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0err =3D _rtnl_link_get_kern=
el(sock, index, name,
> > > > byref(link))
> > > > 18:56:15 ctypes.ArgumentError: argument 3: <class
> > > > 'TypeError'>: wrong
> > > type
> > > >=20
> > > > 18:56:15 -------------------- >> begin captured logging
<<
> > > --------------------
> > > >=20
> > > > 18:56:15 2016-09-20 18:55:55,220 WARNING [py.warnings]
> > > > (MainThread)
> > > > /usr/lib/python3.5/site-packages/nose/util.py:453:
> > > > DeprecationWarning:
> > > > inspect.getargspec() is deprecated, use inspect.signature()
> > > > instead
> > > > 18:56:15=C2=A0=C2=A0=C2=A0inspect.getargspec(func)
> > > > 18:56:15=C2=A0=C2=A0(inspect:1041)
> > > > 18:56:15 2016-09-20 18:55:55,222 INFO=C2=A0=C2=A0[root] (MainThre=
ad)
> > > > Bond
> > > > check_pkopm has been created. (bond:142)
> > > > 18:56:15 2016-09-20 18:55:55,233 INFO=C2=A0=C2=A0[root] (MainThre=
ad)
> > > > Bond
> > > > check_pkopm has been destroyed. (bond:149)
> > > > 18:56:15 2016-09-20 18:55:55,234 DEBUG [root] (MainThread)
> > > > /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name
> > > > dummy_gmayN
> > > > type dummy (cwd None) (commands:69)
> > > > 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread)
> > > > SUCCESS:
> > > > <err> =3D b''; <rc> =3D 0 (commands:93)
> > > > 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread)
> > > > /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name
> > > > dummy_p3yZR
> > > > type dummy (cwd None) (commands:69)
> > > > 18:56:15 2016-09-20 18:55:55,250 DEBUG [root] (MainThread)
> > > > SUCCESS:
> > > > <err> =3D b''; <rc> =3D 0 (commands:93)
> > > > 18:56:15 2016-09-20 18:55:55,258 INFO=C2=A0=C2=A0[root] (MainThre=
ad)
> > > > Bond
> > > > bond_Siiakr has been created. (bond:142)
> > > > 18:56:15 2016-09-20 18:55:55,265 INFO=C2=A0=C2=A0[root] (MainThre=
ad)
> > > > Bond
> > > > bond_Siiakr has been destroyed. (bond:149)
> > > > 18:56:15 2016-09-20 18:55:55,265 DEBUG [root] (MainThread)
> > > > /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev
> > > > dummy_gmayN (cwd
> > > > None) (commands:69)
> > > > 18:56:15 2016-09-20 18:55:55,277 DEBUG [root] (MainThread)
> > > > SUCCESS:
> > > > <err> =3D b''; <rc> =3D 0 (commands:93)
> > > > 18:56:15 2016-09-20 18:55:55,278 DEBUG [root] (MainThread)
> > > > /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev
> > > > dummy_p3yZR (cwd
> > > > None) (commands:69)
> > > > 18:56:15 2016-09-20 18:55:55,289 DEBUG [root] (MainThread)
> > > > SUCCESS:
> > > > <err> =3D b''; <rc> =3D 0 (commands:93)
> > > > 18:56:15 --------------------- >> end captured logging
<<
> > > ---------------------
> > >=20
> >=20
> >=20
--=-67cwno0plYCf4UiDPw0o
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: This is a digitally signed message part
Content-Transfer-Encoding: 7bit
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAABCAAGBQJX4/geAAoJECnCNm5N/FcoWxkP/j8ivxektJKmnfO3Ca5VCtgP
gBzE1P5URuDgHEtdrom3HV/NT9fGFVosB4xZmPOjmtqgLmhwrf7cBr/ZqWu2DV5P
0RCbKILImo31puX3qECHXPCbv9yo5W9cW0qsnIbHDCqxNGy+UukC0YsfCjMZ2OGO
rKVN3zmrVxHZHoizmO9OZ8AjzeKPxz64UFHiAlavljp4Zkj8wv/rzT7d5g40vqCM
aqrTVgASBvt16rvbOdWU3ZY4rmUsiE8cByRShsjJ84NMlaumW3CG0ohvEXVrg/MQ
G+TDFUL6YF0Coicw2kw/v3Wu5y73T6rVDPTCcUOBkQ6AZOA3VjekbcNsTT5JU7+z
pL3u1L2k4DMomUKzTHiBkFoik2eaV3fwLtsCZXzW3VR+ddSfztB2fn5UR68hQsik
pnNmLr7dERmQPu/BZr8KGkpJTdI51DVNaq62u8/kxm1TsXd2/Bo9kkkvSCLEqtgQ
a2zeHkoE1Yh3BPnPM+bg0kYPfNn2aL3hVB/KbuQBkA8ivEZ4jL5hxUh8t+/2Ci4/
y99JpIVd6Z3S3tym003MVGjXNc0cMkItxk/faThy9PpyQD+0RnwCccMJKWW3bkGV
O6Ve8Qnidzh9nUY4AUpba6lbL4N7V0Ha/9nL7lSy8vHkOpaisLCtjWVUxysmIK+1
svplWXnhrVLFLZbIs29N
=X39r
-----END PGP SIGNATURE-----
--=-67cwno0plYCf4UiDPw0o--
5:33 p.m.
And here:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2337/con...
On Thu, Sep 22, 2016 at 5:34 PM, Piotr Kliczewski <pkliczew(a)redhat.com>
wrote:
On Thu, Sep 22, 2016 at 9:58 AM, Dan Kenigsberg <danken(a)redhat.com> wrote:
> On Wed, Sep 21, 2016 at 12:18:37AM +0300, Nir Soffer wrote:
> > On Tue, Sep 20, 2016 at 8:18 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
> > > Failed only one patch when building 4, so this looks like
> > > a random failure, but the error is disturbing.
>
> Petr, could you look into it? Could it be that libnl3 changed their API?
> Or that a slave is pulling an ancient lib? Or broken kernel? Nothing
> makes sense.
>
> > >
> > > See http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x8
> 6_64/2255/console
> > >
> > > 17:00:31 ============================================================
> ==========
> > > 17:00:31 ERROR: test_bond_create_failure_on_slave_add
> > > (network.link_bond_test.LinkBondTests)
> > > 17:00:31 ------------------------------------------------------------
> ----------
> > > 17:00:31 Traceback (most recent call last):
> > > 17:00:31 File
> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/tests/network/link_bond_test.py",
> > > line 83, in test_bond_create_failure_on_slave_add
> > > 17:00:31 base_bond.add_slaves((nic1, nic2))
> > > 17:00:31 File
> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/link/bond.py",
> > > line 153, in add_slaves
> > > 17:00:31 with _preserve_iface_state(slave):
> > > 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59,
in
> __enter__
> > > 17:00:31 return next(self.gen)
> > > 17:00:31 File
> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/link/bond.py",
> > > line 214, in _preserve_iface_state
> > > 17:00:31 dev_was_up = iface.is_up(dev)
> > > 17:00:31 File
> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/link/iface.py",
> > > line 54, in is_up
> > > 17:00:31 return is_admin_up(dev)
> > > 17:00:31 File
> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/link/iface.py",
> > > line 58, in is_admin_up
> > > 17:00:31 return is_link_up(get_link(dev)['flags'],
> check_oper_status=False)
> > > 17:00:31 File
> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/netlink/link.py",
> > > line 56, in get_link
> > > 17:00:31 with _get_link(name=name, sock=sock) as link:
> > > 17:00:31 File "/usr/lib64/python3.5/contextlib.py", line 59,
in
> __enter__
> > > 17:00:31 return next(self.gen)
> > > 17:00:31 File
> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/netlink/link.py",
> > > line 199, in _get_link
> > > 17:00:31 err = _rtnl_link_get_kernel(sock, index, name,
> byref(link))
> > > 17:00:31 ctypes.ArgumentError: argument 3: <class
'TypeError'>: wrong
> type
> > > 17:00:31 -------------------- >> begin captured logging <<
> --------------------
> > > 17:00:31 2016-09-20 17:00:12,586 WARNING [py.warnings] (MainThread)
> > > /usr/lib/python3.5/site-packages/nose/util.py:453:
> DeprecationWarning:
> > > inspect.getargspec() is deprecated, use inspect.signature() instead
> > > 17:00:31 inspect.getargspec(func)
> > > 17:00:31 (inspect:1041)
>
> Piotr, the DeprecationWarning is unrelated to our code but we are using
> this API, too.
>
>
Please take a look at https://gerrit.ovirt.org/64313
> > > 17:00:31 2016-09-20 17:00:12,587 INFO [root] (MainThread) Bond
> > > check_7lzvb has been created. (bond:142)
> > > 17:00:31 2016-09-20 17:00:12,598 INFO [root] (MainThread) Bond
> > > check_7lzvb has been destroyed. (bond:149)
> > > 17:00:31 2016-09-20 17:00:12,598 DEBUG [root] (MainThread)
> > > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_Dmwc5
> > > type dummy (cwd None) (commands:69)
> > > 17:00:31 2016-09-20 17:00:12,607 DEBUG [root] (MainThread) SUCCESS:
> > > <err> = b''; <rc> = 0 (commands:93)
> > > 17:00:31 2016-09-20 17:00:12,608 DEBUG [root] (MainThread)
> > > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_xIK5C
> > > type dummy (cwd None) (commands:69)
> > > 17:00:31 2016-09-20 17:00:12,615 DEBUG [root] (MainThread) SUCCESS:
> > > <err> = b''; <rc> = 0 (commands:93)
> > > 17:00:31 2016-09-20 17:00:12,617 INFO [root] (MainThread) Bond
> > > bond_rwTTFu has been created. (bond:142)
> > > 17:00:31 2016-09-20 17:00:12,634 INFO [root] (MainThread) Bond
> > > bond_rwTTFu has been destroyed. (bond:149)
> > > 17:00:31 2016-09-20 17:00:12,634 DEBUG [root] (MainThread)
> > > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_Dmwc5
> > > (cwd None) (commands:69)
> > > 17:00:31 2016-09-20 17:00:12,658 DEBUG [root] (MainThread) SUCCESS:
> > > <err> = b''; <rc> = 0 (commands:93)
> > > 17:00:31 2016-09-20 17:00:12,659 DEBUG [root] (MainThread)
> > > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_xIK5C
> > > (cwd None) (commands:69)
> > > 17:00:31 2016-09-20 17:00:12,676 DEBUG [root] (MainThread) SUCCESS:
> > > <err> = b''; <rc> = 0 (commands:93)
> > > 17:00:31 --------------------- >> end captured logging <<
> ---------------------
> >
> > Another instance, again only one patch failed when building 5 patches
> series.
> >
> > This time with new log format for the tests.
> >
> > 18:56:15 ============================================================
> ==========
> > 18:56:15 ERROR: test_bond_create_failure_on_slave_add
> > (network.link_bond_test.LinkBondTests)
> > 18:56:15 ------------------------------------------------------------
> ----------
> > 18:56:15 Traceback (most recent call last):
> > 18:56:15 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/tests/network/link_bond_test.py",
> > line 83, in test_bond_create_failure_on_slave_add
> > 18:56:15 base_bond.add_slaves((nic1, nic2))
> > 18:56:15 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/link/bond.py",
> > line 153, in add_slaves
> > 18:56:15 with _preserve_iface_state(slave):
> > 18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in
> __enter__
> > 18:56:15 return next(self.gen)
> > 18:56:15 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/link/bond.py",
> > line 214, in _preserve_iface_state
> > 18:56:15 dev_was_up = iface.is_up(dev)
> > 18:56:15 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/link/iface.py",
> > line 54, in is_up
> > 18:56:15 return is_admin_up(dev)
> > 18:56:15 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/link/iface.py",
> > line 58, in is_admin_up
> > 18:56:15 return is_link_up(get_link(dev)['flags'],
> check_oper_status=False)
> > 18:56:15 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/netlink/link.py",
> > line 56, in get_link
> > 18:56:15 with _get_link(name=name, sock=sock) as link:
> > 18:56:15 File "/usr/lib64/python3.5/contextlib.py", line 59, in
> __enter__
> > 18:56:15 return next(self.gen)
> > 18:56:15 File
> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> /vdsm/lib/vdsm/network/netlink/link.py",
> > line 199, in _get_link
> > 18:56:15 err = _rtnl_link_get_kernel(sock, index, name, byref(link))
> > 18:56:15 ctypes.ArgumentError: argument 3: <class 'TypeError'>:
wrong
> type
> > 18:56:15 -------------------- >> begin captured logging <<
> --------------------
> > 18:56:15 2016-09-20 18:55:55,220 WARNING [py.warnings] (MainThread)
> > /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning:
> > inspect.getargspec() is deprecated, use inspect.signature() instead
> > 18:56:15 inspect.getargspec(func)
> > 18:56:15 (inspect:1041)
> > 18:56:15 2016-09-20 18:55:55,222 INFO [root] (MainThread) Bond
> > check_pkopm has been created. (bond:142)
> > 18:56:15 2016-09-20 18:55:55,233 INFO [root] (MainThread) Bond
> > check_pkopm has been destroyed. (bond:149)
> > 18:56:15 2016-09-20 18:55:55,234 DEBUG [root] (MainThread)
> > /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_gmayN
> > type dummy (cwd None) (commands:69)
> > 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) SUCCESS:
> > <err> = b''; <rc> = 0 (commands:93)
> > 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread)
> > /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_p3yZR
> > type dummy (cwd None) (commands:69)
> > 18:56:15 2016-09-20 18:55:55,250 DEBUG [root] (MainThread) SUCCESS:
> > <err> = b''; <rc> = 0 (commands:93)
> > 18:56:15 2016-09-20 18:55:55,258 INFO [root] (MainThread) Bond
> > bond_Siiakr has been created. (bond:142)
> > 18:56:15 2016-09-20 18:55:55,265 INFO [root] (MainThread) Bond
> > bond_Siiakr has been destroyed. (bond:149)
> > 18:56:15 2016-09-20 18:55:55,265 DEBUG [root] (MainThread)
> > /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_gmayN (cwd
> > None) (commands:69)
> > 18:56:15 2016-09-20 18:55:55,277 DEBUG [root] (MainThread) SUCCESS:
> > <err> = b''; <rc> = 0 (commands:93)
> > 18:56:15 2016-09-20 18:55:55,278 DEBUG [root] (MainThread)
> > /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_p3yZR (cwd
> > None) (commands:69)
> > 18:56:15 2016-09-20 18:55:55,289 DEBUG [root] (MainThread) SUCCESS:
> > <err> = b''; <rc> = 0 (commands:93)
> > 18:56:15 --------------------- >> end captured logging <<
> ---------------------
>
3018
days inactive
3020
days old
7 comments
4 participants
participants (4)
-
Dan Kenigsberg -
Nir Soffer -
Piotr Kliczewski -
Thomas Haller