<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Sep 22, 2016 at 9:58 AM, Dan Kenigsberg <span dir="ltr"><<a target="_blank" href="mailto:danken@redhat.com">danken@redhat.com</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">On Wed, Sep 21, 2016 at 12:18:37AM +0300, Nir Soffer wrote:<br>
> On Tue, Sep 20, 2016 at 8:18 PM, Nir Soffer <<a href="mailto:nsoffer@redhat.com">nsoffer@redhat.com</a>> wrote:<br>
> > Failed only one patch when building 4, so this looks like<br>
> > a random failure, but the error is disturbing.<br>
<br>
Petr, could you look into it? Could it be that libnl3 changed their API?<br>
Or that a slave is pulling an ancient lib? Or broken kernel? Nothing<br>
makes sense.<br>
<br>
> ><br>
> > See <a target="_blank" rel="noreferrer" href="http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2255/console">http://jenkins.ovirt.org/job/<wbr>vdsm_master_check-patch-fc24-<wbr>x86_64/2255/console</a><br>
> ><br>
> > 17:00:31 ==============================<wbr>==============================<wbr>==========<br>
> > 17:00:31 ERROR: test_bond_create_failure_on_<wbr>slave_add<br>
> > (network.link_bond_test.<wbr>LinkBondTests)<br>
> > 17:00:31 ------------------------------<wbr>------------------------------<wbr>----------<br>
> > 17:00:31 Traceback (most recent call last):<br>
> > 17:00:31 File<br>
> > "/home/jenkins/workspace/vdsm_<wbr>master_check-patch-fc24-x86_<wbr>64/vdsm/tests/network/link_<wbr>bond_test.py",<br>
> > line 83, in test_bond_create_failure_on_<wbr>slave_add<br>
> > 17:00:31 base_bond.add_slaves((nic1, nic2))<br>
> > 17:00:31 File<br>
> > "/home/jenkins/workspace/vdsm_<wbr>master_check-patch-fc24-x86_<wbr>64/vdsm/lib/vdsm/network/link/<wbr>bond.py",<br>
> > line 153, in add_slaves<br>
> > 17:00:31 with _preserve_iface_state(slave):<br>
> > 17:00:31 File "/usr/lib64/python3.5/<wbr>contextlib.py", line 59, in __enter__<br>
> > 17:00:31 return next(self.gen)<br>
> > 17:00:31 File<br>
> > "/home/jenkins/workspace/vdsm_<wbr>master_check-patch-fc24-x86_<wbr>64/vdsm/lib/vdsm/network/link/<wbr>bond.py",<br>
> > line 214, in _preserve_iface_state<br>
> > 17:00:31 dev_was_up = iface.is_up(dev)<br>
> > 17:00:31 File<br>
> > "/home/jenkins/workspace/vdsm_<wbr>master_check-patch-fc24-x86_<wbr>64/vdsm/lib/vdsm/network/link/<wbr>iface.py",<br>
> > line 54, in is_up<br>
> > 17:00:31 return is_admin_up(dev)<br>
> > 17:00:31 File<br>
> > "/home/jenkins/workspace/vdsm_<wbr>master_check-patch-fc24-x86_<wbr>64/vdsm/lib/vdsm/network/link/<wbr>iface.py",<br>
> > line 58, in is_admin_up<br>
> > 17:00:31 return is_link_up(get_link(dev)['<wbr>flags'], check_oper_status=False)<br>
> > 17:00:31 File<br>
> > "/home/jenkins/workspace/vdsm_<wbr>master_check-patch-fc24-x86_<wbr>64/vdsm/lib/vdsm/network/<wbr>netlink/link.py",<br>
> > line 56, in get_link<br>
> > 17:00:31 with _get_link(name=name, sock=sock) as link:<br>
> > 17:00:31 File "/usr/lib64/python3.5/<wbr>contextlib.py", line 59, in __enter__<br>
> > 17:00:31 return next(self.gen)<br>
> > 17:00:31 File<br>
> > "/home/jenkins/workspace/vdsm_<wbr>master_check-patch-fc24-x86_<wbr>64/vdsm/lib/vdsm/network/<wbr>netlink/link.py",<br>
> > line 199, in _get_link<br>
> > 17:00:31 err = _rtnl_link_get_kernel(sock, index, name, byref(link))<br>
> > 17:00:31 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type<br>
> > 17:00:31 -------------------- >> begin captured logging << --------------------<br>
> > 17:00:31 2016-09-20 17:00:12,586 WARNING [py.warnings] (MainThread)<br>
> > /usr/lib/python3.5/site-<wbr>packages/nose/util.py:453: DeprecationWarning:<br>
> > inspect.getargspec() is deprecated, use inspect.signature() instead<br>
> > 17:00:31 inspect.getargspec(func)<br>
> > 17:00:31 (inspect:1041)<br>
<br>
Piotr, the DeprecationWarning is unrelated to our code but we are using<br>
this API, too.<br>
<br></blockquote><div><br></div><div>Please take a look at <a href="https://gerrit.ovirt.org/64313">https://gerrit.ovirt.org/64313</a><br></div><div> </div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
> > 17:00:31 2016-09-20 17:00:12,587 INFO [root] (MainThread) Bond<br>
> > check_7lzvb has been created. (bond:142)<br>
> > 17:00:31 2016-09-20 17:00:12,598 INFO [root] (MainThread) Bond<br>
> > check_7lzvb has been destroyed. (bond:149)<br>
> > 17:00:31 2016-09-20 17:00:12,598 DEBUG [root] (MainThread)<br>
> > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_Dmwc5<br>
> > type dummy (cwd None) (commands:69)<br>
> > 17:00:31 2016-09-20 17:00:12,607 DEBUG [root] (MainThread) SUCCESS:<br>
> > <err> = b''; <rc> = 0 (commands:93)<br>
> > 17:00:31 2016-09-20 17:00:12,608 DEBUG [root] (MainThread)<br>
> > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_xIK5C<br>
> > type dummy (cwd None) (commands:69)<br>
> > 17:00:31 2016-09-20 17:00:12,615 DEBUG [root] (MainThread) SUCCESS:<br>
> > <err> = b''; <rc> = 0 (commands:93)<br>
> > 17:00:31 2016-09-20 17:00:12,617 INFO [root] (MainThread) Bond<br>
> > bond_rwTTFu has been created. (bond:142)<br>
> > 17:00:31 2016-09-20 17:00:12,634 INFO [root] (MainThread) Bond<br>
> > bond_rwTTFu has been destroyed. (bond:149)<br>
> > 17:00:31 2016-09-20 17:00:12,634 DEBUG [root] (MainThread)<br>
> > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_Dmwc5<br>
> > (cwd None) (commands:69)<br>
> > 17:00:31 2016-09-20 17:00:12,658 DEBUG [root] (MainThread) SUCCESS:<br>
> > <err> = b''; <rc> = 0 (commands:93)<br>
> > 17:00:31 2016-09-20 17:00:12,659 DEBUG [root] (MainThread)<br>
> > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_xIK5C<br>
> > (cwd None) (commands:69)<br>
> > 17:00:31 2016-09-20 17:00:12,676 DEBUG [root] (MainThread) SUCCESS:<br>
> > <err> = b''; <rc> = 0 (commands:93)<br>
> > 17:00:31 --------------------- >> end captured logging << ---------------------<br>
><br>
> Another instance, again only one patch failed when building 5 patches series.<br>
><br>
> This time with new log format for the tests.<br>
><br>
> 18:56:15 ==============================<wbr>==============================<wbr>==========<br>
> 18:56:15 ERROR: test_bond_create_failure_on_<wbr>slave_add<br>
> (network.link_bond_test.<wbr>LinkBondTests)<br>
> 18:56:15 ------------------------------<wbr>------------------------------<wbr>----------<br>
> 18:56:15 Traceback (most recent call last):<br>
> 18:56:15 File<br>
> "/home/jenkins/workspace/vdsm_<wbr>master_check-patch-fc24-x86_<wbr>64/vdsm/tests/network/link_<wbr>bond_test.py",<br>
> line 83, in test_bond_create_failure_on_<wbr>slave_add<br>
> 18:56:15 base_bond.add_slaves((nic1, nic2))<br>
> 18:56:15 File<br>
> "/home/jenkins/workspace/vdsm_<wbr>master_check-patch-fc24-x86_<wbr>64/vdsm/lib/vdsm/network/link/<wbr>bond.py",<br>
> line 153, in add_slaves<br>
> 18:56:15 with _preserve_iface_state(slave):<br>
> 18:56:15 File "/usr/lib64/python3.5/<wbr>contextlib.py", line 59, in __enter__<br>
> 18:56:15 return next(self.gen)<br>
> 18:56:15 File<br>
> "/home/jenkins/workspace/vdsm_<wbr>master_check-patch-fc24-x86_<wbr>64/vdsm/lib/vdsm/network/link/<wbr>bond.py",<br>
> line 214, in _preserve_iface_state<br>
> 18:56:15 dev_was_up = iface.is_up(dev)<br>
> 18:56:15 File<br>
> "/home/jenkins/workspace/vdsm_<wbr>master_check-patch-fc24-x86_<wbr>64/vdsm/lib/vdsm/network/link/<wbr>iface.py",<br>
> line 54, in is_up<br>
> 18:56:15 return is_admin_up(dev)<br>
> 18:56:15 File<br>
> "/home/jenkins/workspace/vdsm_<wbr>master_check-patch-fc24-x86_<wbr>64/vdsm/lib/vdsm/network/link/<wbr>iface.py",<br>
> line 58, in is_admin_up<br>
> 18:56:15 return is_link_up(get_link(dev)['<wbr>flags'], check_oper_status=False)<br>
> 18:56:15 File<br>
> "/home/jenkins/workspace/vdsm_<wbr>master_check-patch-fc24-x86_<wbr>64/vdsm/lib/vdsm/network/<wbr>netlink/link.py",<br>
> line 56, in get_link<br>
> 18:56:15 with _get_link(name=name, sock=sock) as link:<br>
> 18:56:15 File "/usr/lib64/python3.5/<wbr>contextlib.py", line 59, in __enter__<br>
> 18:56:15 return next(self.gen)<br>
> 18:56:15 File<br>
> "/home/jenkins/workspace/vdsm_<wbr>master_check-patch-fc24-x86_<wbr>64/vdsm/lib/vdsm/network/<wbr>netlink/link.py",<br>
> line 199, in _get_link<br>
> 18:56:15 err = _rtnl_link_get_kernel(sock, index, name, byref(link))<br>
> 18:56:15 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type<br>
> 18:56:15 -------------------- >> begin captured logging << --------------------<br>
> 18:56:15 2016-09-20 18:55:55,220 WARNING [py.warnings] (MainThread)<br>
> /usr/lib/python3.5/site-<wbr>packages/nose/util.py:453: DeprecationWarning:<br>
> inspect.getargspec() is deprecated, use inspect.signature() instead<br>
> 18:56:15 inspect.getargspec(func)<br>
> 18:56:15 (inspect:1041)<br>
> 18:56:15 2016-09-20 18:55:55,222 INFO [root] (MainThread) Bond<br>
> check_pkopm has been created. (bond:142)<br>
> 18:56:15 2016-09-20 18:55:55,233 INFO [root] (MainThread) Bond<br>
> check_pkopm has been destroyed. (bond:149)<br>
> 18:56:15 2016-09-20 18:55:55,234 DEBUG [root] (MainThread)<br>
> /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_gmayN<br>
> type dummy (cwd None) (commands:69)<br>
> 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) SUCCESS:<br>
> <err> = b''; <rc> = 0 (commands:93)<br>
> 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread)<br>
> /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_p3yZR<br>
> type dummy (cwd None) (commands:69)<br>
> 18:56:15 2016-09-20 18:55:55,250 DEBUG [root] (MainThread) SUCCESS:<br>
> <err> = b''; <rc> = 0 (commands:93)<br>
> 18:56:15 2016-09-20 18:55:55,258 INFO [root] (MainThread) Bond<br>
> bond_Siiakr has been created. (bond:142)<br>
> 18:56:15 2016-09-20 18:55:55,265 INFO [root] (MainThread) Bond<br>
> bond_Siiakr has been destroyed. (bond:149)<br>
> 18:56:15 2016-09-20 18:55:55,265 DEBUG [root] (MainThread)<br>
> /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_gmayN (cwd<br>
> None) (commands:69)<br>
> 18:56:15 2016-09-20 18:55:55,277 DEBUG [root] (MainThread) SUCCESS:<br>
> <err> = b''; <rc> = 0 (commands:93)<br>
> 18:56:15 2016-09-20 18:55:55,278 DEBUG [root] (MainThread)<br>
> /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_p3yZR (cwd<br>
> None) (commands:69)<br>
> 18:56:15 2016-09-20 18:55:55,289 DEBUG [root] (MainThread) SUCCESS:<br>
> <err> = b''; <rc> = 0 (commands:93)<br>
> 18:56:15 --------------------- >> end captured logging << ---------------------<br>
</blockquote></div><br></div></div>