Re: All vdsm builds fail now

Wednesday, 16 January 2019

On Tue, Jan 15, 2019 at 9:01 PM Nir Soffer <nsoffer(a)redhat.com&gt; wrote:

...
 See https://jenkins.ovirt.org/job/vdsm_standard-check-patch/

https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_...

 Both fail in network tests that used to work yesterday.

 Is this https://gerrit.ovirt.org/c/96921/?

Yes. We have not seen these fail with that patch.
It just got merged, so lets examine the results from now on.

We may have a problem/regression with fc29. We should spawn a job for it
soon.

...

======================================================================
 ERROR: test_multiple_vlans (network.tc_test.TestConfigureOutbound)
 ----------------------------------------------------------------------
 Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
     return func(*args, **keywargs)
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/network/tc_test.py",
line 455, in test_multiple_vlans
     self.device_name, v.tag)
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/configurators/qos.py",
line 47, in configure_outbound
     _fresh_qdisc_conf_out(device, vlan_tag, class_id, qosOutbound)
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/configurators/qos.py",
line 116, in _fresh_qdisc_conf_out
     default='%#x' % _NON_VLANNED_ID)
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/tc/qdisc.py",
line 43, in add
     _wrapper.process_request(command)
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/tc/_wrapper.py",
line 43, in process_request
     raise TrafficControlException(retcode, err, command)
 TrafficControlException: (34, 'RTNETLINK answers: Numerical result out of range',
['/sbin/tc', 'qdisc', 'add', 'dev', 'dummy_ghyx4',
'root', 'handle', '0x1389:', 'hfsc', 'default',
'0x1388'])
 -------------------- >> begin captured logging << --------------------
 2019-01-15 18:27:00,089 DEBUG (MainThread) [root] /sbin/ip link add name dummy_ghyx4 type
dummy (cwd None) (cmdutils:133)
 2019-01-15 18:27:00,105 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,107 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_ghyx4 up
(cwd None) (cmdutils:133)
 2019-01-15 18:27:00,135 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,138 DEBUG (MainThread) [root] /sbin/ip link add link dummy_ghyx4 name
dummy_ghyx4.16 type vlan id 16 (cwd None) (cmdutils:133)
 2019-01-15 18:27:00,159 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,163 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_ghyx4.16 up
(cwd None) (cmdutils:133)
 2019-01-15 18:27:00,193 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,196 DEBUG (MainThread) [root] /sbin/ip link add link dummy_ghyx4 name
dummy_ghyx4.17 type vlan id 17 (cwd None) (cmdutils:133)
 2019-01-15 18:27:00,216 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,218 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_ghyx4.17 up
(cwd None) (cmdutils:133)
 2019-01-15 18:27:00,236 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,238 DEBUG (MainThread) [root] /sbin/tc qdisc show dev dummy_ghyx4
(cwd None) (cmdutils:133)
 2019-01-15 18:27:00,275 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,282 DEBUG (MainThread) [root] /sbin/tc qdisc del dev dummy_ghyx4 root
(cwd None) (cmdutils:133)
 2019-01-15 18:27:00,305 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK
answers: No such file or directory\n'; <rc> = 2 (cmdutils:141)
 2019-01-15 18:27:00,307 DEBUG (MainThread) [root] /sbin/tc qdisc del dev dummy_ghyx4
ingress (cwd None) (cmdutils:133)
 2019-01-15 18:27:00,320 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK
answers: No such file or directory\n'; <rc> = 2 (cmdutils:141)
 2019-01-15 18:27:00,323 DEBUG (MainThread) [root] /sbin/tc qdisc add dev dummy_ghyx4 root
handle 0x1389: hfsc default 0x1388 (cwd None) (cmdutils:133)
 2019-01-15 18:27:00,350 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK
answers: Numerical result out of range\n'; <rc> = 2 (cmdutils:141)
 2019-01-15 18:27:00,355 DEBUG (netlink/events) [root] START thread
<Thread(netlink/events, started daemon 140628604352256)> (func=<bound method
Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at
0x7fe6a7a1b0d0>>, args=(), kwargs={}) (concurrent:193)
 2019-01-15 18:27:00,358 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_ghyx4.17
down (cwd None) (cmdutils:133)
 2019-01-15 18:27:00,376 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,378 DEBUG (netlink/events) [root] FINISH thread
<Thread(netlink/events, started daemon 140628604352256)> (concurrent:196)
 2019-01-15 18:27:00,380 DEBUG (MainThread) [root] /sbin/ip link del dev dummy_ghyx4.17
(cwd None) (cmdutils:133)
 2019-01-15 18:27:00,399 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,404 DEBUG (netlink/events) [root] START thread
<Thread(netlink/events, started daemon 140628604352256)> (func=<bound method
Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at
0x7fe6a7a1b110>>, args=(), kwargs={}) (concurrent:193)
 2019-01-15 18:27:00,407 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_ghyx4.16
down (cwd None) (cmdutils:133)
 2019-01-15 18:27:00,423 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,425 DEBUG (netlink/events) [root] FINISH thread
<Thread(netlink/events, started daemon 140628604352256)> (concurrent:196)
 2019-01-15 18:27:00,425 DEBUG (MainThread) [root] /sbin/ip link del dev dummy_ghyx4.16
(cwd None) (cmdutils:133)
 2019-01-15 18:27:00,447 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 --------------------- >> end captured logging << ---------------------

 ======================================================================
 ERROR: test_single_non_vlan (network.tc_test.TestConfigureOutbound)
 ----------------------------------------------------------------------
 Traceback (most recent call last):
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/network/tc_test.py",
line 416, in test_single_non_vlan
     qos.configure_outbound(HOST_QOS_OUTBOUND, self.device_name, None)
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/configurators/qos.py",
line 47, in configure_outbound
     _fresh_qdisc_conf_out(device, vlan_tag, class_id, qosOutbound)
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/configurators/qos.py",
line 116, in _fresh_qdisc_conf_out
     default='%#x' % _NON_VLANNED_ID)
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/tc/qdisc.py",
line 43, in add
     _wrapper.process_request(command)
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/tc/_wrapper.py",
line 43, in process_request
     raise TrafficControlException(retcode, err, command)
 TrafficControlException: (34, 'RTNETLINK answers: Numerical result out of range',
['/sbin/tc', 'qdisc', 'add', 'dev', 'dummy_ErG82',
'root', 'handle', '0x1389:', 'hfsc', 'default',
'0x1388'])
 -------------------- >> begin captured logging << --------------------
 2019-01-15 18:27:00,477 DEBUG (MainThread) [root] /sbin/ip link add name dummy_ErG82 type
dummy (cwd None) (cmdutils:133)
 2019-01-15 18:27:00,488 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,490 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_ErG82 up
(cwd None) (cmdutils:133)
 2019-01-15 18:27:00,501 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,502 DEBUG (MainThread) [root] /sbin/tc qdisc show dev dummy_ErG82
(cwd None) (cmdutils:133)
 2019-01-15 18:27:00,516 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,517 DEBUG (MainThread) [root] /sbin/tc qdisc del dev dummy_ErG82 root
(cwd None) (cmdutils:133)
 2019-01-15 18:27:00,529 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK
answers: No such file or directory\n'; <rc> = 2 (cmdutils:141)
 2019-01-15 18:27:00,531 DEBUG (MainThread) [root] /sbin/tc qdisc del dev dummy_ErG82
ingress (cwd None) (cmdutils:133)
 2019-01-15 18:27:00,541 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK
answers: No such file or directory\n'; <rc> = 2 (cmdutils:141)
 2019-01-15 18:27:00,542 DEBUG (MainThread) [root] /sbin/tc qdisc add dev dummy_ErG82 root
handle 0x1389: hfsc default 0x1388 (cwd None) (cmdutils:133)
 2019-01-15 18:27:00,552 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK
answers: Numerical result out of range\n'; <rc> = 2 (cmdutils:141)
 --------------------- >> end captured logging << ---------------------

 ======================================================================
 ERROR: test_single_vlan(1) (network.tc_test.TestConfigureOutbound)
 ----------------------------------------------------------------------
 Traceback (most recent call last):
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line
142, in wrapper
     return f(self, *args)
   File "/usr/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
     return func(*args, **keywargs)
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/network/tc_test.py",
line 434, in test_single_vlan
     vlan.tag)
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/configurators/qos.py",
line 47, in configure_outbound
     _fresh_qdisc_conf_out(device, vlan_tag, class_id, qosOutbound)
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/configurators/qos.py",
line 116, in _fresh_qdisc_conf_out
     default='%#x' % _NON_VLANNED_ID)
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/tc/qdisc.py",
line 43, in add
     _wrapper.process_request(command)
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/tc/_wrapper.py",
line 43, in process_request
     raise TrafficControlException(retcode, err, command)
 TrafficControlException: (34, 'RTNETLINK answers: Numerical result out of range',
['/sbin/tc', 'qdisc', 'add', 'dev', 'dummy_RLG36',
'root', 'handle', '0x1389:', 'hfsc', 'default',
'0x1388'])
 -------------------- >> begin captured logging << --------------------
 2019-01-15 18:27:00,578 DEBUG (MainThread) [root] /sbin/ip link add name dummy_RLG36 type
dummy (cwd None) (cmdutils:133)
 2019-01-15 18:27:00,587 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,589 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_RLG36 up
(cwd None) (cmdutils:133)
 2019-01-15 18:27:00,601 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,603 DEBUG (MainThread) [root] /sbin/ip link add link dummy_RLG36 name
dummy_RLG36.16 type vlan id 16 (cwd None) (cmdutils:133)
 2019-01-15 18:27:00,613 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,614 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_RLG36.16 up
(cwd None) (cmdutils:133)
 2019-01-15 18:27:00,641 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,642 DEBUG (MainThread) [root] /sbin/tc qdisc show dev dummy_RLG36
(cwd None) (cmdutils:133)
 2019-01-15 18:27:00,660 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,661 DEBUG (MainThread) [root] /sbin/tc qdisc del dev dummy_RLG36 root
(cwd None) (cmdutils:133)
 2019-01-15 18:27:00,670 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK
answers: No such file or directory\n'; <rc> = 2 (cmdutils:141)
 2019-01-15 18:27:00,672 DEBUG (MainThread) [root] /sbin/tc qdisc del dev dummy_RLG36
ingress (cwd None) (cmdutils:133)
 2019-01-15 18:27:00,681 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK
answers: No such file or directory\n'; <rc> = 2 (cmdutils:141)
 2019-01-15 18:27:00,683 DEBUG (MainThread) [root] /sbin/tc qdisc add dev dummy_RLG36 root
handle 0x1389: hfsc default 0x1388 (cwd None) (cmdutils:133)
 2019-01-15 18:27:00,693 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK
answers: Numerical result out of range\n'; <rc> = 2 (cmdutils:141)
 2019-01-15 18:27:00,695 DEBUG (netlink/events) [root] START thread
<Thread(netlink/events, started daemon 140628604352256)> (func=<bound method
Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at
0x7fe6a7b33b90>>, args=(), kwargs={}) (concurrent:193)
 2019-01-15 18:27:00,697 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_RLG36.16
down (cwd None) (cmdutils:133)
 2019-01-15 18:27:00,712 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,714 DEBUG (netlink/events) [root] FINISH thread
<Thread(netlink/events, started daemon 140628604352256)> (concurrent:196)
 2019-01-15 18:27:00,715 DEBUG (MainThread) [root] /sbin/ip link del dev dummy_RLG36.16
(cwd None) (cmdutils:133)
 2019-01-15 18:27:00,729 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 --------------------- >> end captured logging << ---------------------

 ======================================================================
 ERROR: test_single_vlan(2) (network.tc_test.TestConfigureOutbound)
 ----------------------------------------------------------------------
 Traceback (most recent call last):
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line
142, in wrapper
     return f(self, *args)
   File "/usr/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
     return func(*args, **keywargs)
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/network/tc_test.py",
line 434, in test_single_vlan
     vlan.tag)
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/configurators/qos.py",
line 47, in configure_outbound
     _fresh_qdisc_conf_out(device, vlan_tag, class_id, qosOutbound)
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/configurators/qos.py",
line 116, in _fresh_qdisc_conf_out
     default='%#x' % _NON_VLANNED_ID)
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/tc/qdisc.py",
line 43, in add
     _wrapper.process_request(command)
   File
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/tc/_wrapper.py",
line 43, in process_request
     raise TrafficControlException(retcode, err, command)
 TrafficControlException: (34, 'RTNETLINK answers: Numerical result out of range',
['/sbin/tc', 'qdisc', 'add', 'dev', 'dummy_0a4K1',
'root', 'handle', '0x1389:', 'hfsc', 'default',
'0x1388'])
 -------------------- >> begin captured logging << --------------------
 2019-01-15 18:27:00,752 DEBUG (MainThread) [root] /sbin/ip link add name dummy_0a4K1 type
dummy (cwd None) (cmdutils:133)
 2019-01-15 18:27:00,768 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,769 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_0a4K1 up
(cwd None) (cmdutils:133)
 2019-01-15 18:27:00,782 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,784 DEBUG (MainThread) [root] /sbin/ip link add link dummy_0a4K1 name
dummy_0a4K1.16 type vlan id 16 (cwd None) (cmdutils:133)
 2019-01-15 18:27:00,803 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,805 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_0a4K1.16 up
(cwd None) (cmdutils:133)
 2019-01-15 18:27:00,824 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,826 DEBUG (MainThread) [root] /sbin/tc qdisc show dev dummy_0a4K1
(cwd None) (cmdutils:133)
 2019-01-15 18:27:00,846 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,848 DEBUG (MainThread) [root] /sbin/tc qdisc del dev dummy_0a4K1 root
(cwd None) (cmdutils:133)
 2019-01-15 18:27:00,858 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK
answers: No such file or directory\n'; <rc> = 2 (cmdutils:141)
 2019-01-15 18:27:00,859 DEBUG (MainThread) [root] /sbin/tc qdisc del dev dummy_0a4K1
ingress (cwd None) (cmdutils:133)
 2019-01-15 18:27:00,869 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK
answers: No such file or directory\n'; <rc> = 2 (cmdutils:141)
 2019-01-15 18:27:00,870 DEBUG (MainThread) [root] /sbin/tc qdisc add dev dummy_0a4K1 root
handle 0x1389: hfsc default 0x1388 (cwd None) (cmdutils:133)
 2019-01-15 18:27:00,889 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK
answers: Numerical result out of range\n'; <rc> = 2 (cmdutils:141)
 2019-01-15 18:27:00,893 DEBUG (netlink/eveCoverage.py warning: Module
/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/vdsm was never imported.
(module-not-imported)
 nts) [root] START thread <Thread(netlink/events, started daemon 140628604352256)>
(func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at
0x7fe6a7b33d10>>, args=(), kwargs={}) (concurrent:193)
 2019-01-15 18:27:00,895 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_0a4K1.16
down (cwd None) (cmdutils:133)
 2019-01-15 18:27:00,909 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 2019-01-15 18:27:00,912 DEBUG (netlink/events) [root] FINISH thread
<Thread(netlink/events, started daemon 140628604352256)> (concurrent:196)
 2019-01-15 18:27:00,913 DEBUG (MainThread) [root] /sbin/ip link del dev dummy_0a4K1.16
(cwd None) (cmdutils:133)
 2019-01-15 18:27:00,930 DEBUG (MainThread) [root] SUCCESS: <err> = '';
<rc> = 0 (cmdutils:141)
 --------------------- >> end captured logging << ---------------------

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Re: All vdsm builds fail now