[ovirt-devel] Vdsm tests are 4X times faster on travis

[Nir Soffer]

On Sun, Dec 4, 2016 at 9:59 AM, Barak Korren <bkorren at redhat.com> wrote:
> On 3 December 2016 at 21:36, Nir Soffer <nsoffer at redhat.com> wrote:
>> HI all,
>>
>> Watching vdsm travis builds in the last weeks, it is clear that vdsm tests
>> on travis are about 4X times faster compared with jenkins builds.
>>
>> Here is a typical build:
>>
>> ovirt ci: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/5101/consoleFull
>> travis ci: https://travis-ci.org/nirs/vdsm/builds/179056079
>>
>> The build took 4:34 on travis, and 19:34 on ovirt ci.
>
> Interesting, thanks for looking at this!
>
>>
>> This has a huge impact on vdsm maintainers. Having to wait 20 minutes
>> for each patch
>> means that we must ignore the ci and merge and hope that previous tests without
>> rebasing on master were good enough.
>>
>> The builds are mostly the same, expect:
>>
>> - In travis we don't check if the build system was changed and
>> packages should be built
>>   takes 9:18 minutes in ovirt ci.
>
> Well, I guess the infra team can't help with that, but still, is there
> anything we could do at the infrastructure level to speed this up?

The line taking 9 minutes is:

    if git diff-tree --no-commit-id --name-only -r HEAD | egrep
--quiet 'vdsm.spec.in|Makefile.am' ; then
        ./automation/build-artifacts.sh
        yum -y install "$EXPORT_DIR/"!(*.src).rpm
    fi

In the build above the condition is false, and we not run
build-artifacts.sh or installing the packages.
The time is spent in:

    git diff-tree --no-commit-id --name-only -r HEAD | egrep --quiet
'vdsm.spec.in|Makefile.am'

Running locally:

$ time git diff-tree --no-commit-id --name-only -r HEAD | egrep
--quiet 'vdsm.spec.in|Makefile.am'

real 0m0.009s
user 0m0.006s
sys 0m0.009s

To debug this we need to get a shell on a jenkins slave with the exact
environment of
a running job.

>
>> - In travis we don't clean or install anything before the test, we use
>> a container with all the
>>   available packages, pulled from dockerhub.
>>   takes about 3:52 minutes in ovirt ci
>
> Well, I guess this is where we (the infra team) should pay attention.
> We do have a plan to switch from mock to Docker at some point
> (OVIRT-873 [1]), but it'll take a while until we can make such a large
> switch.
>
> It the meantime there may be some low-hanging fruit we can pick to
> make things faster. Looking at the same log:
>
> 16:03:28 Init took 77 seconds
>
> 16:05:50 Install packages took 142 seconds
>
> We may be able to speed those up - looking at the way muck is
> configured, we may be running with its caches turned off (I'm not yet
> 100% sure about this - muck_runner.sh is not the simplest script...).
> I've created OVIRT-902 [2] for us to look at this.
>
>> - In travis we don't enable coverage. Running the tests with coverage
>> may slow down the tests
>>   takes 5:04 minutes in ovirt ci
>>   creating the coverage report takes only 15 seconds, not interesting
>
> We can easily check this by just sending a patch with coverage turned
> on and then sending another patch set for the same patch with coverage
> turned off.
>
>> - In travis we don't cleanup anything after the test
>>   this takes 34 seconds in ovirt ci
>
> We can look at speeding this up - or perhaps just change things so
> that results are reported as soon as check_patch.sh is done as opposed
> to when the Jenkins job is done.
> There may be some pitfalls here so I need to think a little more
> before I recommend going down this path.
>
>> The biggest problem is the build system check taking 9:18 minutes.
>> fixing it will cut the build time in half.
>
> Please try fixing that, or maybe this should just move to build_artifacts.sh?
>
> [1]: https://ovirt-jira.atlassian.net/browse/OVIRT-873
> [2]: https://ovirt-jira.atlassian.net/browse/OVIRT-902
>
> --
> Barak Korren
> bkorren at redhat.com
> RHCE, RHCi, RHV-DevOps Team
> https://ifireball.wordpress.com/