Hi Nir, The test scans all bond options for all modes. It does it by modifying all the bond options values and checking the result. We saw the same problem before and fixed it by recreating the bond device per mode. The problem did not reappeared until now. We suspect it is a bond module issue/timing. Has it happen to you more than once? We could try something like this: If it fails on EBUSY, redo the scan from the mode that failed. (hoping that removing and adding the bond will resolve it) Thanks, Edy. ----- Original Message ----- | From: "Nir Soffer" <nsoffer(a)redhat.com&gt; | To: "devel" <devel(a)ovirt.org&gt;, "Dan Kenigsberg" <danken(a)redhat.com&gt;, "Edward Haas" <edwardh(a)redhat.com&gt;, "infra" | <infra(a)ovirt.org&gt; | Sent: Monday, January 4, 2016 7:19:04 PM | Subject: New network test failure on jenkins | | Hi all, | | There is a new networking test failure in the ci: | | 16:18:11 | ====================================================================== | 16:18:11 ERROR: test_dump_bonding_name2numeric | (toolBondingTests.TestToolBonding) | 16:18:11 | ---------------------------------------------------------------------- | 16:18:11 Traceback (most recent call last): | 16:18:11 File | "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testValidation.py", | line 97, in wrapper | 16:18:11 return f(*args, **kwargs) | 16:18:11 File | "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/modprobe.py", | line 58, in wrapper | 16:18:11 return f(*args, **kwargs) | 16:18:11 File | "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/toolBondingTests.py", | line 37, in test_dump_bonding_name2numeric | 16:18:11 opt_map = _get_bonding_options_name2numeric() | 16:18:11 File | "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/vdsm/tool/dump_bonding_opts.py", | line 94, in _get_bonding_options_name2numeric | 16:18:11 opts[mode] = bond_opts_name2numeric_filtered(bond_name) | 16:18:11 File | "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/vdsm/netinfo/bonding.py", | line 203, in bond_opts_name2numeric_filtered | 16:18:11 in _bond_opts_name2numeric(bond).iteritems() | 16:18:11 File | "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/vdsm/netinfo/bonding.py", | line 238, in _bond_opts_name2numeric | 16:18:11 _bond_opts_name2numeric_scan(path) | 16:18:11 File | "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/vdsm/netinfo/bonding.py", | line 247, in _bond_opts_name2numeric_scan | 16:18:11 numeric_val) | 16:18:11 File | "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/vdsm/netinfo/bonding.py", | line 259, in _bond_opts_name2numeric_getval | 16:18:11 opt_write_file.flush() | 16:18:11 IOError: [Errno 16] Device or resource busy: | 'opt[/sys/class/net/Z7PxqN9gLALvKkE/bonding/lacp_rate], | numeric_val[1]' | | See | http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/1831/con... | | Can someone from networking check this? | | Thanks, | Nir |

----- Original Message ----- | From: "Nir Soffer" <nsoffer(a)redhat.com&gt; | To: "Edward Haas" <edwardh(a)redhat.com&gt; | Cc: "devel" <devel(a)ovirt.org&gt;, "Dan Kenigsberg" <danken(a)redhat.com&gt;, "infra" <infra(a)ovirt.org&gt; | Sent: Tuesday, January 5, 2016 12:12:21 AM | Subject: Re: New network test failure on jenkins | | On Mon, Jan 4, 2016 at 11:58 PM, Edward Haas <edwardh(a)redhat.com&gt; wrote: | > Hi Nir, | > | > The test scans all bond options for all modes. | | Does it test our code or the kernel? Our code is tested. | | > It does it by modifying all the bond options values and checking the | > result. | > | > We saw the same problem before and fixed it by recreating | > the bond device per mode. The problem did not reappeared until now. | > We suspect it is a bond module issue/timing. | | If this test depends on timing, mark it as @slowtest, so it will not | run in the ci. But it is not slow, its a bond issue. Dan recommended to skip the test if it fails on EBUSY and remark about the limitation we saw + the kernel version running. | | > Has it happen to you more than once? | | Only once in jenkins so far. | | > We could try something like this: If it fails on EBUSY, | | Do we expect EBUSY? do we handle it in the real code? We do not expect it, as no other entity should touch it in parallel. The tested code is not running in the service, it is manually called to create a bond options map. | | > redo the scan | > from the mode that failed. (hoping that removing and adding the bond | > will resolve it) | > | > Thanks, | > Edy. | > | > ----- Original Message ----- | > | From: "Nir Soffer" <nsoffer(a)redhat.com&gt; | > | To: "devel" <devel(a)ovirt.org&gt;, "Dan Kenigsberg" <danken(a)redhat.com&gt;, | > | "Edward Haas" <edwardh(a)redhat.com&gt;, "infra" | > | <infra(a)ovirt.org&gt; | > | Sent: Monday, January 4, 2016 7:19:04 PM | > | Subject: New network test failure on jenkins | > | | > | Hi all, | > | | > | There is a new networking test failure in the ci: | > | | > | 16:18:11 | > | ====================================================================== | > | 16:18:11 ERROR: test_dump_bonding_name2numeric | > | (toolBondingTests.TestToolBonding) | > | 16:18:11 | > | ---------------------------------------------------------------------- | > | 16:18:11 Traceback (most recent call last): | > | 16:18:11 File | > | "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testValidation.py", | > | line 97, in wrapper | > | 16:18:11 return f(*args, **kwargs) | > | 16:18:11 File | > | "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/modprobe.py", | > | line 58, in wrapper | > | 16:18:11 return f(*args, **kwargs) | > | 16:18:11 File | > | "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/toolBondingTests.py", | > | line 37, in test_dump_bonding_name2numeric | > | 16:18:11 opt_map = _get_bonding_options_name2numeric() | > | 16:18:11 File | > | "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/vdsm/tool/dump_bonding_opts.py", | > | line 94, in _get_bonding_options_name2numeric | > | 16:18:11 opts[mode] = bond_opts_name2numeric_filtered(bond_name) | > | 16:18:11 File | > | "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/vdsm/netinfo/bonding.py", | > | line 203, in bond_opts_name2numeric_filtered | > | 16:18:11 in _bond_opts_name2numeric(bond).iteritems() | > | 16:18:11 File | > | "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/vdsm/netinfo/bonding.py", | > | line 238, in _bond_opts_name2numeric | > | 16:18:11 _bond_opts_name2numeric_scan(path) | > | 16:18:11 File | > | "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/vdsm/netinfo/bonding.py", | > | line 247, in _bond_opts_name2numeric_scan | > | 16:18:11 numeric_val) | > | 16:18:11 File | > | "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/vdsm/netinfo/bonding.py", | > | line 259, in _bond_opts_name2numeric_getval | > | 16:18:11 opt_write_file.flush() | > | 16:18:11 IOError: [Errno 16] Device or resource busy: | > | 'opt[/sys/class/net/Z7PxqN9gLALvKkE/bonding/lacp_rate], | > | numeric_val[1]' | > | | > | See | > | http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/1831/con... | > | | > | Can someone from networking check this? | > | | > | Thanks, | > | Nir | > | |