[ovirt-devel] Vdsm tests are 4X times faster on travis

[Barak Korren]

On 3 December 2016 at 21:36, Nir Soffer <nsoffer at redhat.com> wrote:
> HI all,
>
> Watching vdsm travis builds in the last weeks, it is clear that vdsm tests
> on travis are about 4X times faster compared with jenkins builds.
>
> Here is a typical build:
>
> ovirt ci: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/5101/consoleFull
> travis ci: https://travis-ci.org/nirs/vdsm/builds/179056079
>
> The build took 4:34 on travis, and 19:34 on ovirt ci.

Interesting, thanks for looking at this!

>
> This has a huge impact on vdsm maintainers. Having to wait 20 minutes
> for each patch
> means that we must ignore the ci and merge and hope that previous tests without
> rebasing on master were good enough.
>
> The builds are mostly the same, expect:
>
> - In travis we don't check if the build system was changed and
> packages should be built
>   takes 9:18 minutes in ovirt ci.

Well, I guess the infra team can't help with that, but still, is there
anything we could do at the infrastructure level to speed this up?

> - In travis we don't clean or install anything before the test, we use
> a container with all the
>   available packages, pulled from dockerhub.
>   takes about 3:52 minutes in ovirt ci

Well, I guess this is where we (the infra team) should pay attention.
We do have a plan to switch from mock to Docker at some point
(OVIRT-873 [1]), but it'll take a while until we can make such a large
switch.

It the meantime there may be some low-hanging fruit we can pick to
make things faster. Looking at the same log:

16:03:28 Init took 77 seconds

16:05:50 Install packages took 142 seconds

We may be able to speed those up - looking at the way muck is
configured, we may be running with its caches turned off (I'm not yet
100% sure about this - muck_runner.sh is not the simplest script...).
I've created OVIRT-902 [2] for us to look at this.

> - In travis we don't enable coverage. Running the tests with coverage
> may slow down the tests
>   takes 5:04 minutes in ovirt ci
>   creating the coverage report takes only 15 seconds, not interesting

We can easily check this by just sending a patch with coverage turned
on and then sending another patch set for the same patch with coverage
turned off.

> - In travis we don't cleanup anything after the test
>   this takes 34 seconds in ovirt ci

We can look at speeding this up - or perhaps just change things so
that results are reported as soon as check_patch.sh is done as opposed
to when the Jenkins job is done.
There may be some pitfalls here so I need to think a little more
before I recommend going down this path.

> The biggest problem is the build system check taking 9:18 minutes.
> fixing it will cut the build time in half.

Please try fixing that, or maybe this should just move to build_artifacts.sh?

[1]: https://ovirt-jira.atlassian.net/browse/OVIRT-873
[2]: https://ovirt-jira.atlassian.net/browse/OVIRT-902

-- 
Barak Korren
bkorren at redhat.com
RHCE, RHCi, RHV-DevOps Team
https://ifireball.wordpress.com/