[ovirt-devel] [VDSM] New network test failure - ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type

Thu Sep 22 15:14:48 UTC 2016

On Thu, Sep 22, 2016 at 05:38:48PM +0300, Nir Soffer wrote:
> I think we should get rid of the ctypes code, it is very hard to
> understand, unsafe, and slow.
> 
> If we need to call C we should use proper python extensions.

Correct.
https://trello.com/c/HtmyG8kk/74-use-python-libnl3-instead-of-our-own-ctype-based-code
added to our trello.

Tomas, does python-libnl3 have bindings to handle netlink events? I'm
looking for an equivalent to our netlink monitor
https://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=lib/vdsm/network/netlink/monitor.py;h=380e5c93d513467e82f6a345fb976d75a61a36bc;hb=HEAD

> 
> On Thu, Sep 22, 2016 at 5:34 PM, Piotr Kliczewski <pkliczew at redhat.com>
> wrote:
> 
> >
> >
> > On Thu, Sep 22, 2016 at 9:58 AM, Dan Kenigsberg <danken at redhat.com> wrote:
> >
> >> On Wed, Sep 21, 2016 at 12:18:37AM +0300, Nir Soffer wrote:
> >> > On Tue, Sep 20, 2016 at 8:18 PM, Nir Soffer <nsoffer at redhat.com> wrote:
> >> > > Failed only one patch when building 4, so this looks like
> >> > > a random failure, but the error is disturbing.
> >>
> >> Petr, could you look into it? Could it be that libnl3 changed their API?
> >> Or that a slave is pulling an ancient lib? Or broken kernel? Nothing
> >> makes sense.
> >>
> >> > >
> >> > > See http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x8
> >> 6_64/2255/console
> >> > >
> >> > > 17:00:31 ============================================================
> >> ==========
> >> > > 17:00:31 ERROR: test_bond_create_failure_on_slave_add
> >> > > (network.link_bond_test.LinkBondTests)
> >> > > 17:00:31 ------------------------------------------------------------
> >> ----------
> >> > > 17:00:31 Traceback (most recent call last):
> >> > > 17:00:31   File
> >> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> >> /vdsm/tests/network/link_bond_test.py",
> >> > > line 83, in test_bond_create_failure_on_slave_add
> >> > > 17:00:31     base_bond.add_slaves((nic1, nic2))
> >> > > 17:00:31   File
> >> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> >> /vdsm/lib/vdsm/network/link/bond.py",
> >> > > line 153, in add_slaves
> >> > > 17:00:31     with _preserve_iface_state(slave):
> >> > > 17:00:31   File "/usr/lib64/python3.5/contextlib.py", line 59, in
> >> __enter__
> >> > > 17:00:31     return next(self.gen)
> >> > > 17:00:31   File
> >> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> >> /vdsm/lib/vdsm/network/link/bond.py",
> >> > > line 214, in _preserve_iface_state
> >> > > 17:00:31     dev_was_up = iface.is_up(dev)
> >> > > 17:00:31   File
> >> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> >> /vdsm/lib/vdsm/network/link/iface.py",
> >> > > line 54, in is_up
> >> > > 17:00:31     return is_admin_up(dev)
> >> > > 17:00:31   File
> >> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> >> /vdsm/lib/vdsm/network/link/iface.py",
> >> > > line 58, in is_admin_up
> >> > > 17:00:31     return is_link_up(get_link(dev)['flags'],
> >> check_oper_status=False)
> >> > > 17:00:31   File
> >> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> >> /vdsm/lib/vdsm/network/netlink/link.py",
> >> > > line 56, in get_link
> >> > > 17:00:31     with _get_link(name=name, sock=sock) as link:
> >> > > 17:00:31   File "/usr/lib64/python3.5/contextlib.py", line 59, in
> >> __enter__
> >> > > 17:00:31     return next(self.gen)
> >> > > 17:00:31   File
> >> > > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> >> /vdsm/lib/vdsm/network/netlink/link.py",
> >> > > line 199, in _get_link
> >> > > 17:00:31     err = _rtnl_link_get_kernel(sock, index, name,
> >> byref(link))
> >> > > 17:00:31 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong
> >> type
> >> > > 17:00:31 -------------------- >> begin captured logging <<
> >> --------------------
> >> > > 17:00:31 2016-09-20 17:00:12,586 WARNING [py.warnings] (MainThread)
> >> > > /usr/lib/python3.5/site-packages/nose/util.py:453:
> >> DeprecationWarning:
> >> > > inspect.getargspec() is deprecated, use inspect.signature() instead
> >> > > 17:00:31   inspect.getargspec(func)
> >> > > 17:00:31  (inspect:1041)
> >>
> >> Piotr, the DeprecationWarning is unrelated to our code but we are using
> >> this API, too.
> >>
> >>
> > Please take a look at https://gerrit.ovirt.org/64313
> >
> >
> >> > > 17:00:31 2016-09-20 17:00:12,587 INFO  [root] (MainThread) Bond
> >> > > check_7lzvb has been created. (bond:142)
> >> > > 17:00:31 2016-09-20 17:00:12,598 INFO  [root] (MainThread) Bond
> >> > > check_7lzvb has been destroyed. (bond:149)
> >> > > 17:00:31 2016-09-20 17:00:12,598 DEBUG [root] (MainThread)
> >> > > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_Dmwc5
> >> > > type dummy (cwd None) (commands:69)
> >> > > 17:00:31 2016-09-20 17:00:12,607 DEBUG [root] (MainThread) SUCCESS:
> >> > > <err> = b''; <rc> = 0 (commands:93)
> >> > > 17:00:31 2016-09-20 17:00:12,608 DEBUG [root] (MainThread)
> >> > > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link add name dummy_xIK5C
> >> > > type dummy (cwd None) (commands:69)
> >> > > 17:00:31 2016-09-20 17:00:12,615 DEBUG [root] (MainThread) SUCCESS:
> >> > > <err> = b''; <rc> = 0 (commands:93)
> >> > > 17:00:31 2016-09-20 17:00:12,617 INFO  [root] (MainThread) Bond
> >> > > bond_rwTTFu has been created. (bond:142)
> >> > > 17:00:31 2016-09-20 17:00:12,634 INFO  [root] (MainThread) Bond
> >> > > bond_rwTTFu has been destroyed. (bond:149)
> >> > > 17:00:31 2016-09-20 17:00:12,634 DEBUG [root] (MainThread)
> >> > > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_Dmwc5
> >> > > (cwd None) (commands:69)
> >> > > 17:00:31 2016-09-20 17:00:12,658 DEBUG [root] (MainThread) SUCCESS:
> >> > > <err> = b''; <rc> = 0 (commands:93)
> >> > > 17:00:31 2016-09-20 17:00:12,659 DEBUG [root] (MainThread)
> >> > > /usr/bin/taskset --cpu-list 0-15 /sbin/ip link del dev dummy_xIK5C
> >> > > (cwd None) (commands:69)
> >> > > 17:00:31 2016-09-20 17:00:12,676 DEBUG [root] (MainThread) SUCCESS:
> >> > > <err> = b''; <rc> = 0 (commands:93)
> >> > > 17:00:31 --------------------- >> end captured logging <<
> >> ---------------------
> >> >
> >> > Another instance, again only one patch failed when building 5 patches
> >> series.
> >> >
> >> > This time with new log format for the tests.
> >> >
> >> > 18:56:15 ============================================================
> >> ==========
> >> > 18:56:15 ERROR: test_bond_create_failure_on_slave_add
> >> > (network.link_bond_test.LinkBondTests)
> >> > 18:56:15 ------------------------------------------------------------
> >> ----------
> >> > 18:56:15 Traceback (most recent call last):
> >> > 18:56:15   File
> >> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> >> /vdsm/tests/network/link_bond_test.py",
> >> > line 83, in test_bond_create_failure_on_slave_add
> >> > 18:56:15     base_bond.add_slaves((nic1, nic2))
> >> > 18:56:15   File
> >> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> >> /vdsm/lib/vdsm/network/link/bond.py",
> >> > line 153, in add_slaves
> >> > 18:56:15     with _preserve_iface_state(slave):
> >> > 18:56:15   File "/usr/lib64/python3.5/contextlib.py", line 59, in
> >> __enter__
> >> > 18:56:15     return next(self.gen)
> >> > 18:56:15   File
> >> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> >> /vdsm/lib/vdsm/network/link/bond.py",
> >> > line 214, in _preserve_iface_state
> >> > 18:56:15     dev_was_up = iface.is_up(dev)
> >> > 18:56:15   File
> >> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> >> /vdsm/lib/vdsm/network/link/iface.py",
> >> > line 54, in is_up
> >> > 18:56:15     return is_admin_up(dev)
> >> > 18:56:15   File
> >> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> >> /vdsm/lib/vdsm/network/link/iface.py",
> >> > line 58, in is_admin_up
> >> > 18:56:15     return is_link_up(get_link(dev)['flags'],
> >> check_oper_status=False)
> >> > 18:56:15   File
> >> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> >> /vdsm/lib/vdsm/network/netlink/link.py",
> >> > line 56, in get_link
> >> > 18:56:15     with _get_link(name=name, sock=sock) as link:
> >> > 18:56:15   File "/usr/lib64/python3.5/contextlib.py", line 59, in
> >> __enter__
> >> > 18:56:15     return next(self.gen)
> >> > 18:56:15   File
> >> > "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
> >> /vdsm/lib/vdsm/network/netlink/link.py",
> >> > line 199, in _get_link
> >> > 18:56:15     err = _rtnl_link_get_kernel(sock, index, name, byref(link))
> >> > 18:56:15 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong
> >> type
> >> > 18:56:15 -------------------- >> begin captured logging <<
> >> --------------------
> >> > 18:56:15 2016-09-20 18:55:55,220 WARNING [py.warnings] (MainThread)
> >> > /usr/lib/python3.5/site-packages/nose/util.py:453: DeprecationWarning:
> >> > inspect.getargspec() is deprecated, use inspect.signature() instead
> >> > 18:56:15   inspect.getargspec(func)
> >> > 18:56:15  (inspect:1041)
> >> > 18:56:15 2016-09-20 18:55:55,222 INFO  [root] (MainThread) Bond
> >> > check_pkopm has been created. (bond:142)
> >> > 18:56:15 2016-09-20 18:55:55,233 INFO  [root] (MainThread) Bond
> >> > check_pkopm has been destroyed. (bond:149)
> >> > 18:56:15 2016-09-20 18:55:55,234 DEBUG [root] (MainThread)
> >> > /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_gmayN
> >> > type dummy (cwd None) (commands:69)
> >> > 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread) SUCCESS:
> >> > <err> = b''; <rc> = 0 (commands:93)
> >> > 18:56:15 2016-09-20 18:55:55,242 DEBUG [root] (MainThread)
> >> > /usr/bin/taskset --cpu-list 0-3 /sbin/ip link add name dummy_p3yZR
> >> > type dummy (cwd None) (commands:69)
> >> > 18:56:15 2016-09-20 18:55:55,250 DEBUG [root] (MainThread) SUCCESS:
> >> > <err> = b''; <rc> = 0 (commands:93)
> >> > 18:56:15 2016-09-20 18:55:55,258 INFO  [root] (MainThread) Bond
> >> > bond_Siiakr has been created. (bond:142)
> >> > 18:56:15 2016-09-20 18:55:55,265 INFO  [root] (MainThread) Bond
> >> > bond_Siiakr has been destroyed. (bond:149)
> >> > 18:56:15 2016-09-20 18:55:55,265 DEBUG [root] (MainThread)
> >> > /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_gmayN (cwd
> >> > None) (commands:69)
> >> > 18:56:15 2016-09-20 18:55:55,277 DEBUG [root] (MainThread) SUCCESS:
> >> > <err> = b''; <rc> = 0 (commands:93)
> >> > 18:56:15 2016-09-20 18:55:55,278 DEBUG [root] (MainThread)
> >> > /usr/bin/taskset --cpu-list 0-3 /sbin/ip link del dev dummy_p3yZR (cwd
> >> > None) (commands:69)
> >> > 18:56:15 2016-09-20 18:55:55,289 DEBUG [root] (MainThread) SUCCESS:
> >> > <err> = b''; <rc> = 0 (commands:93)
> >> > 18:56:15 --------------------- >> end captured logging <<
> >> ---------------------
> >>
> >
> >