[VDSM] Flaky (networking?) test

Hi all, I have seen this failure several times in the last week, typically the error is gone after rebuilding. It is not clear what is the test - the test name does not apear in the error, this happens when adding a (useless) docstring to a test method. Looking at the traceback, this seems to be a networking test. Full log: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc25-x86_64/218/console... 14:55:03 ====================================================================== 14:55:03 ERROR: Failure: OSError ([Errno 22] Invalid argument) 14:55:03 ---------------------------------------------------------------------- 14:55:03 Traceback (most recent call last): 14:55:03 File "/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_64/vdsm/lib/vdsm/utils.py", line 477, in __call__ 14:55:03 return self.cache[args] 14:55:03 KeyError: () 14:55:03 14:55:03 During handling of the above exception, another exception occurred: 14:55:03 14:55:03 OSError: [Errno 22] Invalid argument 14:55:03 14:55:03 During handling of the above exception, another exception occurred: 14:55:03 14:55:03 Traceback (most recent call last): 14:55:03 File "/usr/lib/python3.5/site-packages/nose/failure.py", line 39, in runTest 14:55:03 raise self.exc_val.with_traceback(self.tb) 14:55:03 File "/usr/lib/python3.5/site-packages/nose/loader.py", line 418, in loadTestsFromName 14:55:03 addr.filename, addr.module) 14:55:03 File "/usr/lib/python3.5/site-packages/nose/importer.py", line 47, in importFromPath 14:55:03 return self.importFromDir(dir_path, fqname) 14:55:03 File "/usr/lib/python3.5/site-packages/nose/importer.py", line 94, in importFromDir 14:55:03 mod = load_module(part_fqname, fh, filename, desc) 14:55:03 File "/usr/lib64/python3.5/imp.py", line 234, in load_module 14:55:03 return load_source(name, filename, file) 14:55:03 File "/usr/lib64/python3.5/imp.py", line 172, in load_source 14:55:03 module = _load(spec) 14:55:03 File "<frozen importlib._bootstrap>", line 693, in _load 14:55:03 File "<frozen importlib._bootstrap>", line 673, in _load_unlocked 14:55:03 File "<frozen importlib._bootstrap_external>", line 665, in exec_module 14:55:03 File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed 14:55:03 File "/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_64/vdsm/tests/network/models_test.py", line 41, in <module> 14:55:03 bonding_default_fpath()) 14:55:03 File "/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_64/vdsm/lib/vdsm/utils.py", line 479, in __call__ 14:55:03 value = self.func(*args) 14:55:03 File "/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_64/vdsm/tests/network/nettestlib.py", line 509, in bonding_default_fpath 14:55:03 dump_bonding_opts() 14:55:03 File "/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_64/vdsm/lib/vdsm/tool/dump_bonding_opts.py", line 55, in main 14:55:03 jdump(_get_bonding_options_name2numeric(), f) 14:55:03 File "/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_64/vdsm/lib/vdsm/tool/dump_bonding_opts.py", line 94, in _get_bonding_options_name2numeric 14:55:03 opts[mode] = bond_opts_name2numeric_filtered(bond_name) 14:55:03 File "/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_64/vdsm/lib/vdsm/network/netinfo/bonding.py", line 154, in bond_opts_name2numeric_filtered 14:55:03 in _bond_opts_name2numeric(bond).iteritems() 14:55:03 File "/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_64/vdsm/lib/vdsm/network/netinfo/bonding.py", line 189, in _bond_opts_name2numeric 14:55:03 _bond_opts_name2numeric_scan(path) 14:55:03 File "/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_64/vdsm/lib/vdsm/network/netinfo/bonding.py", line 202, in _bond_opts_name2numeric_scan 14:55:03 vals[name] = numeric 14:55:03 OSError: [Errno 22] Invalid argument

Rebase on top of master. It was your hint to avoid buffering when opening /sys/class/net/bond/option that I used to fix this py3-only behavior. On Jan 13, 2017 17:51, "Nir Soffer" <nsoffer@redhat.com> wrote:
Hi all,
I have seen this failure several times in the last week, typically the error is gone after rebuilding.
It is not clear what is the test - the test name does not apear in the error, this happens when adding a (useless) docstring to a test method.
Looking at the traceback, this seems to be a networking test.
Full log: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc25- x86_64/218/consoleFull
14:55:03 ============================================================ ========== 14:55:03 ERROR: Failure: OSError ([Errno 22] Invalid argument) 14:55:03 ------------------------------------------------------------ ---------- 14:55:03 Traceback (most recent call last): 14:55:03 File "/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_ 64/vdsm/lib/vdsm/utils.py", line 477, in __call__ 14:55:03 return self.cache[args] 14:55:03 KeyError: () 14:55:03 14:55:03 During handling of the above exception, another exception occurred: 14:55:03 14:55:03 OSError: [Errno 22] Invalid argument 14:55:03 14:55:03 During handling of the above exception, another exception occurred: 14:55:03 14:55:03 Traceback (most recent call last): 14:55:03 File "/usr/lib/python3.5/site-packages/nose/failure.py", line 39, in runTest 14:55:03 raise self.exc_val.with_traceback(self.tb) 14:55:03 File "/usr/lib/python3.5/site-packages/nose/loader.py", line 418, in loadTestsFromName 14:55:03 addr.filename, addr.module) 14:55:03 File "/usr/lib/python3.5/site-packages/nose/importer.py", line 47, in importFromPath 14:55:03 return self.importFromDir(dir_path, fqname) 14:55:03 File "/usr/lib/python3.5/site-packages/nose/importer.py", line 94, in importFromDir 14:55:03 mod = load_module(part_fqname, fh, filename, desc) 14:55:03 File "/usr/lib64/python3.5/imp.py", line 234, in load_module 14:55:03 return load_source(name, filename, file) 14:55:03 File "/usr/lib64/python3.5/imp.py", line 172, in load_source 14:55:03 module = _load(spec) 14:55:03 File "<frozen importlib._bootstrap>", line 693, in _load 14:55:03 File "<frozen importlib._bootstrap>", line 673, in _load_unlocked 14:55:03 File "<frozen importlib._bootstrap_external>", line 665, in exec_module 14:55:03 File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed 14:55:03 File "/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_ 64/vdsm/tests/network/models_test.py", line 41, in <module> 14:55:03 bonding_default_fpath()) 14:55:03 File "/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_ 64/vdsm/lib/vdsm/utils.py", line 479, in __call__ 14:55:03 value = self.func(*args) 14:55:03 File "/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_ 64/vdsm/tests/network/nettestlib.py", line 509, in bonding_default_fpath 14:55:03 dump_bonding_opts() 14:55:03 File "/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_ 64/vdsm/lib/vdsm/tool/dump_bonding_opts.py", line 55, in main 14:55:03 jdump(_get_bonding_options_name2numeric(), f) 14:55:03 File "/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_ 64/vdsm/lib/vdsm/tool/dump_bonding_opts.py", line 94, in _get_bonding_options_name2numeric 14:55:03 opts[mode] = bond_opts_name2numeric_filtered(bond_name) 14:55:03 File "/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_ 64/vdsm/lib/vdsm/network/netinfo/bonding.py", line 154, in bond_opts_name2numeric_filtered 14:55:03 in _bond_opts_name2numeric(bond).iteritems() 14:55:03 File "/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_ 64/vdsm/lib/vdsm/network/netinfo/bonding.py", line 189, in _bond_opts_name2numeric 14:55:03 _bond_opts_name2numeric_scan(path) 14:55:03 File "/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_ 64/vdsm/lib/vdsm/network/netinfo/bonding.py", line 202, in _bond_opts_name2numeric_scan 14:55:03 vals[name] = numeric 14:55:03 OSError: [Errno 22] Invalid argument

Great. Still we should find the failing test and remove the useless docstring that hides the test name in the error. On Fri, Jan 13, 2017 at 5:59 PM, Dan Kenigsberg <danken@redhat.com> wrote:
Rebase on top of master. It was your hint to avoid buffering when opening /sys/class/net/bond/option that I used to fix this py3-only behavior.
On Jan 13, 2017 17:51, "Nir Soffer" <nsoffer@redhat.com> wrote:
Hi all,
I have seen this failure several times in the last week, typically the error is gone after rebuilding.
It is not clear what is the test - the test name does not apear in the error, this happens when adding a (useless) docstring to a test method.
Looking at the traceback, this seems to be a networking test.
Full log:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc25-x86_64/218/console...
14:55:03 ====================================================================== 14:55:03 ERROR: Failure: OSError ([Errno 22] Invalid argument) 14:55:03 ---------------------------------------------------------------------- 14:55:03 Traceback (most recent call last): 14:55:03 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_64/vdsm/lib/vdsm/utils.py", line 477, in __call__ 14:55:03 return self.cache[args] 14:55:03 KeyError: () 14:55:03 14:55:03 During handling of the above exception, another exception occurred: 14:55:03 14:55:03 OSError: [Errno 22] Invalid argument 14:55:03 14:55:03 During handling of the above exception, another exception occurred: 14:55:03 14:55:03 Traceback (most recent call last): 14:55:03 File "/usr/lib/python3.5/site-packages/nose/failure.py", line 39, in runTest 14:55:03 raise self.exc_val.with_traceback(self.tb) 14:55:03 File "/usr/lib/python3.5/site-packages/nose/loader.py", line 418, in loadTestsFromName 14:55:03 addr.filename, addr.module) 14:55:03 File "/usr/lib/python3.5/site-packages/nose/importer.py", line 47, in importFromPath 14:55:03 return self.importFromDir(dir_path, fqname) 14:55:03 File "/usr/lib/python3.5/site-packages/nose/importer.py", line 94, in importFromDir 14:55:03 mod = load_module(part_fqname, fh, filename, desc) 14:55:03 File "/usr/lib64/python3.5/imp.py", line 234, in load_module 14:55:03 return load_source(name, filename, file) 14:55:03 File "/usr/lib64/python3.5/imp.py", line 172, in load_source 14:55:03 module = _load(spec) 14:55:03 File "<frozen importlib._bootstrap>", line 693, in _load 14:55:03 File "<frozen importlib._bootstrap>", line 673, in _load_unlocked 14:55:03 File "<frozen importlib._bootstrap_external>", line 665, in exec_module 14:55:03 File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed 14:55:03 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_64/vdsm/tests/network/models_test.py", line 41, in <module> 14:55:03 bonding_default_fpath()) 14:55:03 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_64/vdsm/lib/vdsm/utils.py", line 479, in __call__ 14:55:03 value = self.func(*args) 14:55:03 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_64/vdsm/tests/network/nettestlib.py", line 509, in bonding_default_fpath 14:55:03 dump_bonding_opts() 14:55:03 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_64/vdsm/lib/vdsm/tool/dump_bonding_opts.py", line 55, in main 14:55:03 jdump(_get_bonding_options_name2numeric(), f) 14:55:03 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_64/vdsm/lib/vdsm/tool/dump_bonding_opts.py", line 94, in _get_bonding_options_name2numeric 14:55:03 opts[mode] = bond_opts_name2numeric_filtered(bond_name) 14:55:03 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_64/vdsm/lib/vdsm/network/netinfo/bonding.py", line 154, in bond_opts_name2numeric_filtered 14:55:03 in _bond_opts_name2numeric(bond).iteritems() 14:55:03 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_64/vdsm/lib/vdsm/network/netinfo/bonding.py", line 189, in _bond_opts_name2numeric 14:55:03 _bond_opts_name2numeric_scan(path) 14:55:03 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_64/vdsm/lib/vdsm/network/netinfo/bonding.py", line 202, in _bond_opts_name2numeric_scan 14:55:03 vals[name] = numeric 14:55:03 OSError: [Errno 22] Invalid argument

On Fri, Jan 13, 2017 at 10:08 PM, Dan Kenigsberg <danken@redhat.com> wrote:
On Fri, Jan 13, 2017 at 6:10 PM, Nir Soffer <nsoffer@redhat.com> wrote:
Great. Still we should find the failing test and remove the useless docstring that hides the test name in the error.
Which docstring do you refer to?
I think this is not a docstring, this seems like an error loading a test module: 14:54:44 network.vm_migrate_hook_test.TestOvsHookMigration 14:54:44 test_legacy_to_legacy_non_vlan OK 14:54:44 test_legacy_to_legacy_with_vlan OK 14:54:44 test_legacy_to_ovs_non_vlan OK 14:54:44 test_legacy_to_ovs_with_vlan OK 14:54:44 test_ovs_to_legacy_non_vlan OK 14:54:44 test_ovs_to_legacy_with_vlan OK 14:54:44 test_ovs_to_ovs_non_vlan OK 14:54:44 test_ovs_to_ovs_with_vlan OK 14:54:44 nose.failure.Failure 14:54:44 runTest ERROR nose does not give any clue about the file that failed, looking in the traceback it seems to be models_test.py. So we are probably ok now. 14:54:44 osinfo_test.TestOsinfo 14:54:44 test_kernel_args(b'', '') OK 14:54:44 test_kernel_args(b'\n', '') OK 14:54:44 test_kernel_args(b'a', 'a') OK 14:54:44 test_kernel_args(b'a\n', 'a') OK 14:54:44 test_kernel_args(b'a\nb', 'a') OK
participants (2)
-
Dan Kenigsberg
-
Nir Soffer