[ovirt-devel] [VDSM] Moving to py.test

[Nir Soffer]

Hi all,

In the last vdsm call we discussed moving to py.test.

Here is how to run the current tests with py.test:

    cd tests
    PYTHONPATH=../lib:../vdsm py.test -v *_test.py *_tests.py *Tests.py network

Currently we have some failures and errors:

    8 failed, 1662 passed, 101 skipped, 14 pytest-warnings, 10 error
in 86.05 seconds

Some py.test nice tricks:

Finding the 10 slowest tests:

    PYTHONPATH=../lib:../vdsm py.test -v --durations=10 vmTests.py
*_test.py *_tests.py *Tests.py network

Checking code coverage:

    PYTHONPATH=../lib:../vdsm py.test -v --cov=../vdsm --cov=vdsm
*_test.py *_tests.py *Tests.py network

Once we finish moving vdsm code to lib/vdsm, renaming the tests to
*_test.py, and add
pytest.ini, running the test will be simply:

    py.test

Once all the tests pass, we can:

- Drop all the special asserts (assertXXX), py.test uses plain asserts
  https://pytest.org/latest/assert.html#asserting-with-the-assert-statement

- Get better error reports
  https://pytest.org/latest/example/reportingdemo.html

- Remove unneeded classes, or inherit from any class
  https://pytest.org/latest/getting-started.html

- Replace @permutations with more powerfull @pytest.mark.parametrize
  https://pytest.org/latest/parametrize.html#parametrize-basics

- Replace slowtest and stresstest with @pytest.mark
  https://pytest.org/latest/mark.html

- Replace monkeypatch.py with simpler and more powerfull pytest
monkeypatch funcarg
  https://pytest.org/latest/monkeypatch.html#monkeypatching-mocking-modules-and-environments

- Replace NamedTemporaryDir with pytest tmpdir funcarg
  https://pytest.org/latest/getting-started.html#going-functional-requesting-a-unique-temporary-directory

- Simplify complex context managers with pytest fixtures
  https://pytest.org/latest/example/simple.html

- Drop testrunner.py, most of testlib.py, some of testValidation.py,
run_tests*.sh

- And more

Nir