Re: [ovirt-devel] New test failure on travis

On 08/12/16 11:26 +0200, Edward Haas wrote: > > On Thu, Dec 8, 2016 at 10:12 AM, Martin Polednik <mpolednik(a)redhat.com&gt; > wrote: > >> On 08/12/16 09:28 +0200, Edward Haas wrote: >> >>> On Wed, Dec 7, 2016 at 11:54 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >>> >>> broken_on_ci is uses default name="OVIRT_CI", to mark it also for >>>> >>>> travis, we need another broken_on_ci with name="TRAVIS_CI". >>>> >>>> Maybe this test should run only if nm is active on the machine? >>>> >>>> >>> We need the test to always run when expected. >>> If NM is not running, the test will not run (silently) and we will never >>> know if there is a problem or not. >>> >>> It is not convenient to mark each CI type as broken, why the test code >>> needs to know we have multiple >>> CI/s? >>> >> >> I believe this is great point - we should just mark the test as broken >> on *any* CI to create a pressure to get it fixed. >> >> Slight off-topic addition: I don't understand why patch marking a test >> as broken on CI takes more than 5 minutes to get merged in when given >> pointer to the failure. > > > > Because it is a wrong approach. :) > If a test fails, it is a smell that something bad happens and we may have > a > production problem. > So before marking and excluding the test, one should feel very guilty that > this check is no longer covered and better understand why it fails. Those are 1-in-N case breakages. The fact the test is unstable should be noted by a maintainer, but shouldn't block any other patches or series (which is what often happens). I don't feel any guilt marking bad test as bad. > >> >> >> Currently, we run on CI tests that are not marked as 'functional'. >>> >>> Perhaps we need another test type that can be mark not to run on simple >>> CI. >>> "power-integration", "super-integration"? >>> >>> >>> >>> >>>> >>>> On Wed, Dec 7, 2016 at 11:23 PM, Dan Kenigsberg <danken(a)redhat.com&gt; >>>> wrote: >>>> > On Wed, Dec 7, 2016 at 2:03 PM, Nir Soffer <nsoffer(a)redhat.com&gt; >>>> > wrote: >>>> >> Looks like we need @brokentest("reason...", name="TRAVIC_CI") on >>>> >> this: >>>> > >>>> > Odd, the code already has >>>> > >>>> > @broken_on_ci('NetworkManager should not be started on CI nodes') >>>> > >>>> > >>>> >> >>>> >> See https://travis-ci.org/oVirt/vdsm/jobs/181933329 >>>> >> >>>> >> ============================================================ >>>> ========== >>>> >> >>>> >> ERROR: test suite for <module 'network.nmdbus_test' from >>>> >> '/vdsm/tests/network/nmdbus_test.py'> >>>> >> >>>> >> ------------------------------------------------------------ >>>> ---------- >>>> >> >>>> >> Traceback (most recent call last): >>>> >> >>>> >> File "/usr/lib/python2.7/site-packages/nose/suite.py", line 209, >>>> >> in >>>> run >>>> >> >>>> >> self.setUp() >>>> >> >>>> >> File "/usr/lib/python2.7/site-packages/nose/suite.py", line 292, >>>> >> in >>>> setUp >>>> >> >>>> >> self.setupContext(ancestor) >>>> >> >>>> >> File "/usr/lib/python2.7/site-packages/nose/suite.py", line 315, >>>> >> in >>>> >> setupContext >>>> >> >>>> >> try_run(context, names) >>>> >> >>>> >> File "/usr/lib/python2.7/site-packages/nose/util.py", line 471, in >>>> try_run >>>> >> >>>> >> return func() >>>> >> >>>> >> File "/vdsm/tests/testValidation.py", line 191, in wrapper >>>> >> >>>> >> return f(*args, **kwargs) >>>> >> >>>> >> File "/vdsm/tests/testValidation.py", line 97, in wrapper >>>> >> >>>> >> return f(*args, **kwargs) >>>> >> >>>> >> File "/vdsm/tests/network/nmdbus_test.py", line 48, in >>>> >> setup_module >>>> >> >>>> >> NMDbus.init() >>>> >> >>>> >> File "/vdsm/lib/vdsm/network/nm/nmdbus/__init__.py", line 33, in >>>> init >>>> >> >>>> >> NMDbus.bus = dbus.SystemBus() >>>> >> >>>> >> File "/usr/lib64/python2.7/site-packages/dbus/_dbus.py", line 194, >>>> in __new__ >>>> >> >>>> >> private=private) >>>> >> >>>> >> File "/usr/lib64/python2.7/site-packages/dbus/_dbus.py", line 100, >>>> in __new__ >>>> >> >>>> >> bus = BusConnection.__new__(subclass, bus_type, >>>> mainloop=mainloop) >>>> >> >>>> >> File "/usr/lib64/python2.7/site-packages/dbus/bus.py", line 122, >>>> >> in >>>> __new__ >>>> >> >>>> >> bus = cls._new_for_bus(address_or_type, mainloop=mainloop) >>>> >> >>>> >> DBusException: org.freedesktop.DBus.Error.FileNotFound: Failed to >>>> >> connect to socket /var/run/dbus/system_bus_socket: No such file or >>>> >> directory >>>> >> >>>> >> -------------------- >> begin captured logging << >>>> >> -------------------- >>>> >> >>>> >> 2016-12-07 11:48:33,458 DEBUG (MainThread) [root] /usr/bin/taskset >>>> >> --cpu-list 0-1 /bin/systemctl status NetworkManager (cwd None) >>>> >> (commands:69) >>>> >> >>>> >> 2016-12-07 11:48:33,465 DEBUG (MainThread) [root] FAILED: <err> = >>>> >> 'Failed to get D-Bus connection: Operation not permitted\n'; <rc> = >>>> >> 1 >>>> >> (commands:93) >>>> >> >>>> >> 2016-12-07 11:48:33,465 DEBUG (MainThread) [root] /usr/bin/taskset >>>> >> --cpu-list 0-1 /bin/systemctl start NetworkManager (cwd None) >>>> >> (commands:69) >>>> >> >>>> >> 2016-12-07 11:48:33,470 DEBUG (MainThread) [root] FAILED: <err> = >>>> >> 'Failed to get D-Bus connection: Operation not permitted\n'; <rc> = >>>> >> 1 >>>> >> (commands:93) >>>> >> >>>> >> --------------------- >> end captured logging << >>>> >> --------------------- >>>> >>>> >> _______________________________________________ >>> >>> Devel mailing list >>> Devel(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/devel >>> >> >>