All vdsm builds fail now ------------------------ Key: OVIRT-2651 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2651 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Nir Soffer Assignee: infra See https://jenkins.ovirt.org/job/vdsm_standard-check-patch/ https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_... https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_... Both fail in network tests that used to work yesterday. Is this https://gerrit.ovirt.org/c/96921/? ====================================================================== ERROR: test_multiple_vlans (network.tc_test.TestConfigureOutbound) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched return func(*args, **keywargs) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/network/tc_test.py", line 455, in test_multiple_vlans self.device_name, v.tag) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/configurators/qos.py", line 47, in configure_outbound _fresh_qdisc_conf_out(device, vlan_tag, class_id, qosOutbound) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/configurators/qos.py", line 116, in _fresh_qdisc_conf_out default='%#x' % _NON_VLANNED_ID) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/tc/qdisc.py", line 43, in add _wrapper.process_request(command) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/tc/_wrapper.py", line 43, in process_request raise TrafficControlException(retcode, err, command) TrafficControlException: (34, 'RTNETLINK answers: Numerical result out of range', ['/sbin/tc', 'qdisc', 'add', 'dev', 'dummy_ghyx4', 'root', 'handle', '0x1389:', 'hfsc', 'default', '0x1388']) -------------------- >> begin captured logging << -------------------- 2019-01-15 18:27:00,089 DEBUG (MainThread) [root] /sbin/ip link add name dummy_ghyx4 type dummy (cwd None) (cmdutils:133) 2019-01-15 18:27:00,105 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,107 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_ghyx4 up (cwd None) (cmdutils:133) 2019-01-15 18:27:00,135 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,138 DEBUG (MainThread) [root] /sbin/ip link add link dummy_ghyx4 name dummy_ghyx4.16 type vlan id 16 (cwd None) (cmdutils:133) 2019-01-15 18:27:00,159 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,163 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_ghyx4.16 up (cwd None) (cmdutils:133) 2019-01-15 18:27:00,193 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,196 DEBUG (MainThread) [root] /sbin/ip link add link dummy_ghyx4 name dummy_ghyx4.17 type vlan id 17 (cwd None) (cmdutils:133) 2019-01-15 18:27:00,216 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,218 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_ghyx4.17 up (cwd None) (cmdutils:133) 2019-01-15 18:27:00,236 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,238 DEBUG (MainThread) [root] /sbin/tc qdisc show dev dummy_ghyx4 (cwd None) (cmdutils:133) 2019-01-15 18:27:00,275 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,282 DEBUG (MainThread) [root] /sbin/tc qdisc del dev dummy_ghyx4 root (cwd None) (cmdutils:133) 2019-01-15 18:27:00,305 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK answers: No such file or directory\n'; <rc> = 2 (cmdutils:141) 2019-01-15 18:27:00,307 DEBUG (MainThread) [root] /sbin/tc qdisc del dev dummy_ghyx4 ingress (cwd None) (cmdutils:133) 2019-01-15 18:27:00,320 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK answers: No such file or directory\n'; <rc> = 2 (cmdutils:141) 2019-01-15 18:27:00,323 DEBUG (MainThread) [root] /sbin/tc qdisc add dev dummy_ghyx4 root handle 0x1389: hfsc default 0x1388 (cwd None) (cmdutils:133) 2019-01-15 18:27:00,350 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK answers: Numerical result out of range\n'; <rc> = 2 (cmdutils:141) 2019-01-15 18:27:00,355 DEBUG (netlink/events) [root] START thread <Thread(netlink/events, started daemon 140628604352256)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fe6a7a1b0d0>>, args=(), kwargs={}) (concurrent:193) 2019-01-15 18:27:00,358 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_ghyx4.17 down (cwd None) (cmdutils:133) 2019-01-15 18:27:00,376 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,378 DEBUG (netlink/events) [root] FINISH thread <Thread(netlink/events, started daemon 140628604352256)> (concurrent:196) 2019-01-15 18:27:00,380 DEBUG (MainThread) [root] /sbin/ip link del dev dummy_ghyx4.17 (cwd None) (cmdutils:133) 2019-01-15 18:27:00,399 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,404 DEBUG (netlink/events) [root] START thread <Thread(netlink/events, started daemon 140628604352256)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fe6a7a1b110>>, args=(), kwargs={}) (concurrent:193) 2019-01-15 18:27:00,407 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_ghyx4.16 down (cwd None) (cmdutils:133) 2019-01-15 18:27:00,423 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,425 DEBUG (netlink/events) [root] FINISH thread <Thread(netlink/events, started daemon 140628604352256)> (concurrent:196) 2019-01-15 18:27:00,425 DEBUG (MainThread) [root] /sbin/ip link del dev dummy_ghyx4.16 (cwd None) (cmdutils:133) 2019-01-15 18:27:00,447 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_single_non_vlan (network.tc_test.TestConfigureOutbound) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/network/tc_test.py", line 416, in test_single_non_vlan qos.configure_outbound(HOST_QOS_OUTBOUND, self.device_name, None) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/configurators/qos.py", line 47, in configure_outbound _fresh_qdisc_conf_out(device, vlan_tag, class_id, qosOutbound) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/configurators/qos.py", line 116, in _fresh_qdisc_conf_out default='%#x' % _NON_VLANNED_ID) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/tc/qdisc.py", line 43, in add _wrapper.process_request(command) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/tc/_wrapper.py", line 43, in process_request raise TrafficControlException(retcode, err, command) TrafficControlException: (34, 'RTNETLINK answers: Numerical result out of range', ['/sbin/tc', 'qdisc', 'add', 'dev', 'dummy_ErG82', 'root', 'handle', '0x1389:', 'hfsc', 'default', '0x1388']) -------------------- >> begin captured logging << -------------------- 2019-01-15 18:27:00,477 DEBUG (MainThread) [root] /sbin/ip link add name dummy_ErG82 type dummy (cwd None) (cmdutils:133) 2019-01-15 18:27:00,488 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,490 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_ErG82 up (cwd None) (cmdutils:133) 2019-01-15 18:27:00,501 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,502 DEBUG (MainThread) [root] /sbin/tc qdisc show dev dummy_ErG82 (cwd None) (cmdutils:133) 2019-01-15 18:27:00,516 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,517 DEBUG (MainThread) [root] /sbin/tc qdisc del dev dummy_ErG82 root (cwd None) (cmdutils:133) 2019-01-15 18:27:00,529 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK answers: No such file or directory\n'; <rc> = 2 (cmdutils:141) 2019-01-15 18:27:00,531 DEBUG (MainThread) [root] /sbin/tc qdisc del dev dummy_ErG82 ingress (cwd None) (cmdutils:133) 2019-01-15 18:27:00,541 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK answers: No such file or directory\n'; <rc> = 2 (cmdutils:141) 2019-01-15 18:27:00,542 DEBUG (MainThread) [root] /sbin/tc qdisc add dev dummy_ErG82 root handle 0x1389: hfsc default 0x1388 (cwd None) (cmdutils:133) 2019-01-15 18:27:00,552 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK answers: Numerical result out of range\n'; <rc> = 2 (cmdutils:141) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_single_vlan(1) (network.tc_test.TestConfigureOutbound) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper return f(self, *args) File "/usr/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched return func(*args, **keywargs) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/network/tc_test.py", line 434, in test_single_vlan vlan.tag) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/configurators/qos.py", line 47, in configure_outbound _fresh_qdisc_conf_out(device, vlan_tag, class_id, qosOutbound) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/configurators/qos.py", line 116, in _fresh_qdisc_conf_out default='%#x' % _NON_VLANNED_ID) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/tc/qdisc.py", line 43, in add _wrapper.process_request(command) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/tc/_wrapper.py", line 43, in process_request raise TrafficControlException(retcode, err, command) TrafficControlException: (34, 'RTNETLINK answers: Numerical result out of range', ['/sbin/tc', 'qdisc', 'add', 'dev', 'dummy_RLG36', 'root', 'handle', '0x1389:', 'hfsc', 'default', '0x1388']) -------------------- >> begin captured logging << -------------------- 2019-01-15 18:27:00,578 DEBUG (MainThread) [root] /sbin/ip link add name dummy_RLG36 type dummy (cwd None) (cmdutils:133) 2019-01-15 18:27:00,587 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,589 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_RLG36 up (cwd None) (cmdutils:133) 2019-01-15 18:27:00,601 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,603 DEBUG (MainThread) [root] /sbin/ip link add link dummy_RLG36 name dummy_RLG36.16 type vlan id 16 (cwd None) (cmdutils:133) 2019-01-15 18:27:00,613 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,614 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_RLG36.16 up (cwd None) (cmdutils:133) 2019-01-15 18:27:00,641 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,642 DEBUG (MainThread) [root] /sbin/tc qdisc show dev dummy_RLG36 (cwd None) (cmdutils:133) 2019-01-15 18:27:00,660 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,661 DEBUG (MainThread) [root] /sbin/tc qdisc del dev dummy_RLG36 root (cwd None) (cmdutils:133) 2019-01-15 18:27:00,670 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK answers: No such file or directory\n'; <rc> = 2 (cmdutils:141) 2019-01-15 18:27:00,672 DEBUG (MainThread) [root] /sbin/tc qdisc del dev dummy_RLG36 ingress (cwd None) (cmdutils:133) 2019-01-15 18:27:00,681 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK answers: No such file or directory\n'; <rc> = 2 (cmdutils:141) 2019-01-15 18:27:00,683 DEBUG (MainThread) [root] /sbin/tc qdisc add dev dummy_RLG36 root handle 0x1389: hfsc default 0x1388 (cwd None) (cmdutils:133) 2019-01-15 18:27:00,693 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK answers: Numerical result out of range\n'; <rc> = 2 (cmdutils:141) 2019-01-15 18:27:00,695 DEBUG (netlink/events) [root] START thread <Thread(netlink/events, started daemon 140628604352256)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fe6a7b33b90>>, args=(), kwargs={}) (concurrent:193) 2019-01-15 18:27:00,697 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_RLG36.16 down (cwd None) (cmdutils:133) 2019-01-15 18:27:00,712 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,714 DEBUG (netlink/events) [root] FINISH thread <Thread(netlink/events, started daemon 140628604352256)> (concurrent:196) 2019-01-15 18:27:00,715 DEBUG (MainThread) [root] /sbin/ip link del dev dummy_RLG36.16 (cwd None) (cmdutils:133) 2019-01-15 18:27:00,729 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_single_vlan(2) (network.tc_test.TestConfigureOutbound) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper return f(self, *args) File "/usr/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched return func(*args, **keywargs) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/network/tc_test.py", line 434, in test_single_vlan vlan.tag) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/configurators/qos.py", line 47, in configure_outbound _fresh_qdisc_conf_out(device, vlan_tag, class_id, qosOutbound) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/configurators/qos.py", line 116, in _fresh_qdisc_conf_out default='%#x' % _NON_VLANNED_ID) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/tc/qdisc.py", line 43, in add _wrapper.process_request(command) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/network/tc/_wrapper.py", line 43, in process_request raise TrafficControlException(retcode, err, command) TrafficControlException: (34, 'RTNETLINK answers: Numerical result out of range', ['/sbin/tc', 'qdisc', 'add', 'dev', 'dummy_0a4K1', 'root', 'handle', '0x1389:', 'hfsc', 'default', '0x1388']) -------------------- >> begin captured logging << -------------------- 2019-01-15 18:27:00,752 DEBUG (MainThread) [root] /sbin/ip link add name dummy_0a4K1 type dummy (cwd None) (cmdutils:133) 2019-01-15 18:27:00,768 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,769 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_0a4K1 up (cwd None) (cmdutils:133) 2019-01-15 18:27:00,782 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,784 DEBUG (MainThread) [root] /sbin/ip link add link dummy_0a4K1 name dummy_0a4K1.16 type vlan id 16 (cwd None) (cmdutils:133) 2019-01-15 18:27:00,803 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,805 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_0a4K1.16 up (cwd None) (cmdutils:133) 2019-01-15 18:27:00,824 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,826 DEBUG (MainThread) [root] /sbin/tc qdisc show dev dummy_0a4K1 (cwd None) (cmdutils:133) 2019-01-15 18:27:00,846 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,848 DEBUG (MainThread) [root] /sbin/tc qdisc del dev dummy_0a4K1 root (cwd None) (cmdutils:133) 2019-01-15 18:27:00,858 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK answers: No such file or directory\n'; <rc> = 2 (cmdutils:141) 2019-01-15 18:27:00,859 DEBUG (MainThread) [root] /sbin/tc qdisc del dev dummy_0a4K1 ingress (cwd None) (cmdutils:133) 2019-01-15 18:27:00,869 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK answers: No such file or directory\n'; <rc> = 2 (cmdutils:141) 2019-01-15 18:27:00,870 DEBUG (MainThread) [root] /sbin/tc qdisc add dev dummy_0a4K1 root handle 0x1389: hfsc default 0x1388 (cwd None) (cmdutils:133) 2019-01-15 18:27:00,889 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK answers: Numerical result out of range\n'; <rc> = 2 (cmdutils:141) 2019-01-15 18:27:00,893 DEBUG (netlink/eveCoverage.py warning: Module /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/vdsm was never imported. (module-not-imported) nts) [root] START thread <Thread(netlink/events, started daemon 140628604352256)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fe6a7b33d10>>, args=(), kwargs={}) (concurrent:193) 2019-01-15 18:27:00,895 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_0a4K1.16 down (cwd None) (cmdutils:133) 2019-01-15 18:27:00,909 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) 2019-01-15 18:27:00,912 DEBUG (netlink/events) [root] FINISH thread <Thread(netlink/events, started daemon 140628604352256)> (concurrent:196) 2019-01-15 18:27:00,913 DEBUG (MainThread) [root] /sbin/ip link del dev dummy_0a4K1.16 (cwd None) (cmdutils:133) 2019-01-15 18:27:00,930 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:141) --------------------- >> end captured logging << ---------------------