On 3 December 2016 at 21:36, Nir Soffer <nsoffer@redhat.com> wrote:
HI all,
Watching vdsm travis builds in the last weeks, it is clear that vdsm tests on travis are about 4X times faster compared with jenkins builds.
Here is a typical build:
ovirt ci: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/5101/consol... travis ci: https://travis-ci.org/nirs/vdsm/builds/179056079
The build took 4:34 on travis, and 19:34 on ovirt ci.
Interesting, thanks for looking at this!
This has a huge impact on vdsm maintainers. Having to wait 20 minutes for each patch means that we must ignore the ci and merge and hope that previous tests without rebasing on master were good enough.
The builds are mostly the same, expect:
- In travis we don't check if the build system was changed and packages should be built takes 9:18 minutes in ovirt ci.
Well, I guess the infra team can't help with that, but still, is there anything we could do at the infrastructure level to speed this up?
- In travis we don't clean or install anything before the test, we use a container with all the available packages, pulled from dockerhub. takes about 3:52 minutes in ovirt ci
Well, I guess this is where we (the infra team) should pay attention. We do have a plan to switch from mock to Docker at some point (OVIRT-873 [1]), but it'll take a while until we can make such a large switch. It the meantime there may be some low-hanging fruit we can pick to make things faster. Looking at the same log: 16:03:28 Init took 77 seconds 16:05:50 Install packages took 142 seconds We may be able to speed those up - looking at the way muck is configured, we may be running with its caches turned off (I'm not yet 100% sure about this - muck_runner.sh is not the simplest script...). I've created OVIRT-902 [2] for us to look at this.
- In travis we don't enable coverage. Running the tests with coverage may slow down the tests takes 5:04 minutes in ovirt ci creating the coverage report takes only 15 seconds, not interesting
We can easily check this by just sending a patch with coverage turned on and then sending another patch set for the same patch with coverage turned off.
- In travis we don't cleanup anything after the test this takes 34 seconds in ovirt ci
We can look at speeding this up - or perhaps just change things so that results are reported as soon as check_patch.sh is done as opposed to when the Jenkins job is done. There may be some pitfalls here so I need to think a little more before I recommend going down this path.
The biggest problem is the build system check taking 9:18 minutes. fixing it will cut the build time in half.
Please try fixing that, or maybe this should just move to build_artifacts.sh? [1]: https://ovirt-jira.atlassian.net/browse/OVIRT-873 [2]: https://ovirt-jira.atlassian.net/browse/OVIRT-902 -- Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/