[ovirt-devel] [vdsm] Running VDSM unit tests on Travis CI using Docker

[Nir Soffer]

On Wed, Apr 6, 2016 at 2:55 PM, Edward Haas <ehaas at redhat.com> wrote:
>
>
> On Wed, Apr 6, 2016 at 2:41 PM, Nir Soffer <nsoffer at redhat.com> wrote:
>>
>> On Wed, Apr 6, 2016 at 2:33 PM, Edward Haas <ehaas at redhat.com> wrote:
>> >
>> >
>> > On Wed, Apr 6, 2016 at 2:23 PM, Nir Soffer <nsoffer at redhat.com> wrote:
>> >>
>> >> On Wed, Apr 6, 2016 at 2:19 PM, Edward Haas <ehaas at redhat.com> wrote:
>> >> >
>> >> >
>> >> > On Wed, Apr 6, 2016 at 1:41 PM, Milan Zamazal <mzamazal at redhat.com>
>> >> > wrote:
>> >> >>
>> >> >> Edward Haas <ehaas at redhat.com> writes:
>> >> >>
>> >> >> > On Wed, Apr 6, 2016 at 11:39 AM, Milan Zamazal
>> >> >> > <mzamazal at redhat.com>
>> >> >> > wrote:
>> >> >> >
>> >> >> >     Thank you, Edward, this is useful not only for CI. I use
>> >> >> > docker
>> >> >> > for
>> >> >> >     building Vdsm and running its unit tests and this helped me to
>> >> >> > get
>> >> >> > the
>> >> >> >     proper updated set of packages after recent changes in Vdsm.
>> >> >> >
>> >> >> >     BTW, it seems that the following packages should be
>> >> >> > additionally
>> >> >> > added
>> >> >> >     for `make check-all': psmisc, which, python-ioprocess
>> >> >> >
>> >> >> >
>> >> >> > Are you saying that make check is passing on your local machine?
>> >> >>
>> >> >> When I add the packages given above, `make check-all' (as well as
>> >> >> `make
>> >> >> check') works for me except for 4 tests in lib/vdsm/schedule.py that
>> >> >> produce the following errors with `make check-all':
>> >> >>
>> >> >>     File "/home/pdm/ovirt/vdsm/vdsm-test/lib/vdsm/schedule.py", line
>> >> >> 134,
>> >> >> in schedule
>> >> >>       heapq.heappush(self._calls, (deadline, call))
>> >> >>   nose.proxy.TypeError: unorderable types: ScheduledCall() <
>> >> >> ScheduledCall()
>> >> >>
>> >> >>     File "/home/pdm/ovirt/vdsm/vdsm-test/tests/scheduleTests.py",
>> >> >> line
>> >> >> 160, in test_latency
>> >> >>       med = ticker.latency[len(ticker.latency) / 2]
>> >> >>   nose.proxy.TypeError: list indices must be integers, not float
>> >> >>
>> >> >> Those are probably Python 3 failures that should be fixed in Vdsm.
>> >> >> The docker environment works fine for running the unit tests on my
>> >> >> machine.
>> >> >
>> >> >
>> >> > I ran it on Travis CI with your recommended addition, and I am
>> >> > getting
>> >> > this
>> >> > result: FAILED (SKIP=107, errors=14):
>> >> > You can view the run here:
>> >> > https://travis-ci.org/EdDev/vdsm/builds/121117253
>> >>
>> >> Sure, make check in master run tests that should not run on travis.
>> >>
>> >> Try the travis branch - after adding ioprocess to the docker image,
>> >> all tests should pass:
>> >> https://gerrit.ovirt.org/55738
>> >>
>> >> Nir
>> >
>> >
>> > Ok, will check it as well.
>> > But its a bit of a lie, many tests are skipped instead of not ran at
>> > all,
>> > this needs to be fixed.
>> > In addition, I find many tests as not unit tests, all tests should pass
>> > in a
>> > few seconds not in 2 minutes.
>>
>> Storage people always lie :-)
>>
>> We have @slowtest for marking slow tests. Unfortunately, some tests are
>> slow, and there is no value in mocking the thing you want to test.
>>
>> Nir
>
>
> Well, I will like to run only the unit tests, and you seem to try and merge
> the integration tests
> in the same run.

make check run only unit tests, I'm using NOSE_EVAL_ATTR to filter out the
integration tests:
https://gerrit.ovirt.org/#/c/55728/2/tests/run_tests_local.sh.in

Some integration tests are not marked yet, we will do this later.

>I think they need to be separated.
> I would like to see the unit tests pass in a few seconds, have no surprising
> skips and
> have no dependency or other services.

I agree all goals, but it will take more time.

We should follow the general rule:
http://c2.com/cgi/wiki?MakeItWorkMakeItRightMakeItFast

Nir