[ovirt-devel] Vdsm tests are 4X times faster on travis

[Nir Soffer]

HI all,

Watching vdsm travis builds in the last weeks, it is clear that vdsm tests
on travis are about 4X times faster compared with jenkins builds.

Here is a typical build:

ovirt ci: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/5101/consoleFull
travis ci: https://travis-ci.org/nirs/vdsm/builds/179056079

The build took 4:34 on travis, and 19:34 on ovirt ci.

This has a huge impact on vdsm maintainers. Having to wait 20 minutes
for each patch
means that we must ignore the ci and merge and hope that previous tests without
rebasing on master were good enough.

The builds are mostly the same, expect:

- In travis we don't check if the build system was changed and
packages should be built
  takes 9:18 minutes in ovirt ci.

- In travis we don't clean or install anything before the test, we use
a container with all the
  available packages, pulled from dockerhub.
  takes about 3:52 minutes in ovirt ci

- In travis we don't enable coverage. Running the tests with coverage
may slow down the tests
  takes 5:04 minutes in ovirt ci
  creating the coverage report takes only 15 seconds, not interesting

- In travis we don't cleanup anything after the test
  this takes 34 seconds in ovirt ci

The biggest problem is the build system check taking 9:18 minutes.
fixing it will cut the build time in half.

This is how time is spent in ovirt ci:

1. Starting (1:28)

00:00:00.001 Triggered by Gerrit: https://gerrit.ovirt.org/67338
00:00:00.039 [EnvInject] - Loading node environment variables.
00:00:00.056 Building remotely on fc24-vm15.phx.ovirt.org (phx nested
local_disk fc24) in workspace
/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64

2. Installing packages (2:24)

00:01:28.338 INFO: installing package(s): autoconf automake gdb git
libguestfs-tools-c libselinux-python3 libvirt-python3 m2crypto make
mom openvswitch policycoreutils-python PyYAML python-blivet
python-coverage python2-decorator python-devel python-inotify
python-ioprocess python-mock python-netaddr python-pthreading
python-setuptools python-six python-requests python3-decorator
python3-netaddr python3-nose python3-six python3-yaml rpm-build
sanlock-python sudo yum yum-utils

3. Setup in check-patch.sh (00:04)

00:03:52.838 + export VDSM_AUTOMATION=1

4. Running the actual tests

00:03:56.670 + ./autogen.sh --system --enable-hooks --enable-vhostmd

5. Installing python-debugifo (00:19)

00:04:15.385 Yum-utils package has been deprecated, use dnf instead.

6. Running make check (05:04)

00:04:30.948 + TIMEOUT=600
00:04:30.948 + make check NOSE_WITH_COVERAGE=1
NOSE_COVER_PACKAGE=/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/vdsm,/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib
...
00:09:33.981   tests: commands succeeded
00:09:33.981   congratulations :)

7. Creating coverage report (00:15)

00:09:34.017 + coverage html -d
/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/exported-artifacts/htmlcov

8. Finding modified files (09:18)

00:09:49.213 + git diff-tree --no-commit-id --name-only -r HEAD
00:09:49.213 + egrep --quiet 'vdsm.spec.in|Makefile.am'

9. Cleaing up (00:27)

00:19:07.994 Took 915 seconds
00:19:34.973 Finished: SUCCESS