Using travis yaml files to specify dependencies and tests

Tue Feb 3 13:04:45 UTC 2015

On 02/03, Sandro Bonazzola wrote:
> Il 20/01/2015 16:50, David Caro ha scritto:
> > 
> > Hi everyone!
> > 
> > After talking a bit with some of you, I think that we can start
> > planning a common build and dependency declaration for tests for the
> > ovirt products, to improve and automate most of the ci process and
> > maintenance.
> > 
> > 
> > 
> > Current status:
> > 
> > == Dependencies
> > 
> > Right now we have 4 types of dependencies:
> > 
> > * test dependencies
> > * tarball/srcrpm build dependencies
> > * rpm build dependencies
> > * installation dependencies
> > 
> > The last two are managed from the spec files through rpm/yum
> > dependency systems. But the first ones are managed manually on the
> > jobs on jenkins or puppet manifests. What separates it from the code
> > that actually requires them and adds an extra layer of maintenance and
> > synchronization between the code, the jenkins jobs and the puppet
> > repository.
> > 
> > 
> > == Builds
> > 
> > We started using autotools to build most of the projects, but it's
> > not a global methodology and even being used on some projects, you
> > need to tune the run for each of them, specifying different variables
> > and running some side scripts.
> > 
> > 
> > == Tests
> > 
> > Some projects use make check to run some of the tests, some tests are
> > totally outside the code and run only in jenkins jobs.
> > 
> > 
> > 
> > Some possible improvements:
> > 
> > == Tests/builds
> > 
> > Using shell scripts:
> > We talked before in another thread to create some generic script to
> > build the artifacts for the product and to run the tests, namely we
> > talked about having 3 executables (bash scripts probably) at the root
> > of each project, that should not require any parameters:
> > 
> >   ./build-artifacts
> >       This should generate any artifacts to be archives (isos, rpms,
> >       debs, tarballs, ...) and leave them at ./exported-artifacts/
> >       directory, for the build system to collect, removing any
> >       previous artifacts if needed.
> > 
> >   ./check_patch
> >      Runs all the tests required for any new patchset in gerrit, for
> >      non-merged changes, should be fast to run to get feedback easily
> > 
> >   ./check_merge
> >      Runs all the tests for any change that is going to be merged
> >      (right now we are not using gates, so it actually after merge,
> >      but the idea is to use this as gate for any merge to the
> >      repo). This can be more resource hungry than check_path.
> > 
> > That way it will let you use easily any framework that you want to use
> > for your project, but still let it be easy to maintain in the global
> > ci/build system (you can use pip, tox, maven, gradle, autotools, make,
> > rake, ...). This will not allow at first running tests in parallel in
> > jenkins, but we can in the future add that possibility (for example,
> > allowing the user to define more than one check script, like
> > check_patch.mytest1 and check_patch.mytest2 and make jenkins run them
> > in parallel).
> > I started a POC of this process here [1]
> > 
> > Using travis yaml files:
> > Using a travis compliant yaml file [2]. That will be less flexible
> > than the above solution and will not allow you to run tests in
> > parallel, though it will let you use travis at any point to offload
> > our ci if needed.
> > 
> > 
> > == Dependencies
> > 
> > Using plain text files:
> > Similar to the above scripts solution, I though of adding an extra
> > file, with the same name, to declare the dependencies to run that
> > script. Adding a suffix in case of different requirements for
> > different distros (matching facter fact strings), for example:
> > 
> >     ./build-artifacts.req
> >         Main requirements file, used if no more specific one
> >         found. With a newline separated list of packages to install on
> >         the environment (jenkins will take care of which package
> >         manager to use).
> > 
> >     ./build-artifacts.req.fc20
> >         Specific requirements for fc20 environment, replaces the
> >         general one if present.
> > 
> > And the same for the other scripts (check_patch and check_merge).
> > 
> > Using travis yaml file:
> > Using a travis compliant yaml file  with some extensions to declare
> > the different dependencies for each type and os/distro. That will
> > allow you to have only one extra file in your repo, though you'd have
> > to duplicate some of the requirements as travis only has ubuntu and
> > forces you to run scripts to install dependencies.
> > 
> > 
> > What do you think? Do you have any better idea?
> 
> 
> I would prefer to have above scripts in something like jenkins, automation or build sub-directory.
> Other than that, no better idea.

As long as all the projects use the same path and name for them, I have no issue
with putting them somewhere else.

I don't think jenkins is a good name, but automation is a nice one.

build is too common, used by a lot of different tools, better avoid it.

So can we agree to use automation directory?

> 
> 
> > 
> > 
> > ps. About using an external repository to store the
> > scripts/requirements for the code. The issue with this is that it
> > forces you to bind a code change in the code repo, to a
> > script/dependency change in the scripts repo, and that adds a  lot of
> > extra maintenance and source of issues and failures. If you know a way
> > of doing it like that without all the fuss, I'd love to hear it.
> > 
> > For example, imagine that you have vdsm and the dependencies are in
> > another repo, now you send a patch to vdsm that requires you to run a
> > specific pep8 version to pass the patch tests, so you have to change
> > the script repo to add that dependency, but doing that you will brake
> > all the other patches tests because they require the older pep8
> > version, so you have to somehow specify in the vdsm patch that you
> > require a specific commit from the scripts repo to be tested with...
> > 
> > Having both in the same repo, allows you to do the code change and the
> > dependency/script change in the same patchset, and test it right away
> > with the correct scripts/deps.
> > 
> > It also binds together code and tests to some point, what is nice to
> > have in a product view, because you know for each version which tests
> > it passed and have a better idea of the possible failures for that
> > version.
> > 
> > 
> > [1] http://gerrit.ovirt.org/#/admin/projects/repoman
> > [2] http://docs.travis-ci.com/
> > 
> > 
> > 
> > _______________________________________________
> > Infra mailing list
> > Infra at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/infra
> > 
> 
> 
> -- 
> Sandro Bonazzola
> Better technology. Faster innovation. Powered by community collaboration.
> See how it works at redhat.com

-- 
David Caro

Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D

Tel.: +420 532 294 605
Email: dcaro at redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20150203/17ea2fa4/attachment.sig>