Hi All, I've been working on adding GitHub Actions-based CI to vdsm today. Feel free to check out the patches here: https://gerrit.ovirt.org/q/topic:%22github-actions-ci%22+(status:open%20OR%2...) Some comments: - the linters work fine already, we can start using them - RPM building works too in general. I think the RPM versions are not right yet, so I'll look into this. After the 'rpm' job is done we get a zipfile with all the built RPMs inside. In the future we may want to run 'createrepo_c' on this dir as well, so we'll have a ready-to-be-used repository in that zip. - 'tests' are working too, but we have a couple of failures we'd need to look at. This job, unlike the others, runs in GitHub's Ubuntu VM inside which we use a privileged container for running the tests. - didn't try 'tests-storage' yet - Not to waist precious free CI minutes and storage, we run linters first, tests after that, but only under the condition that the linters didn't fail, and finally we build RPMs, but this time under the condition that the tests didn't fail. You can find some of the runs I made in my personal fork here: https://github.com/tinez/vdsm/actions/workflows/ci.yml Comments, remarks and reviews are highly appreciated. Regards, Marcin
On Fri, Nov 26, 2021 at 7:44 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi All,
I've been working on adding GitHub Actions-based CI to vdsm today. Feel free to check out the patches here:
https://gerrit.ovirt.org/q/topic:%22github-actions-ci%22+(status:open%20OR%2...)
Awesome!
Some comments: - the linters work fine already, we can start using them
- RPM building works too in general. I think the RPM versions are not right yet, so I'll look into this. After the 'rpm' job is done we get a zipfile with all the built RPMs inside. In the future we may want to run 'createrepo_c' on this dir as well, so we'll have a ready-to-be-used repository in that zip.
We already have code to do this in automation/common.sh - can we reuse it or extract it to new script?
- 'tests' are working too, but we have a couple of failures we'd need to look at. This job, unlike the others, runs in GitHub's Ubuntu VM inside which we use a privileged container for running the tests.
The failure seems to be missing TRAVIS_CI=1 environment variable. This should of course change to something moer generic like CONTAINER_CI=1 and the helper modules tests/testing.py should be updated.
- didn't try 'tests-storage' yet
That's fine, the storage team will handle this.
- Not to waist precious free CI minutes and storage, we run linters first, tests after that, but only under the condition that the linters didn't fail, and finally we build RPMs, but this time under the condition that the tests didn't fail.
This is very bad idea. We want to run all the jobs in parallel so we can get quick feedback from all jobs at the same run. There is no real dependency between the jobs, and we don't waste resources by running them in parallel. Thanks for this work! Nir
On Sun, Nov 28, 2021 at 12:41 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Fri, Nov 26, 2021 at 7:44 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi All,
I've been working on adding GitHub Actions-based CI to vdsm today. Feel free to check out the patches here:
https://gerrit.ovirt.org/q/topic:%22github-actions-ci%22+(status:open%20OR%2...)
Awesome!
Some comments: - the linters work fine already, we can start using them
- RPM building works too in general. I think the RPM versions are not right yet,
rpm versions are corrected by: - pushing current master to your fork - pushing --tags to your fork - using "check-depth: 0" option for actions/checkout@v2
so I'll look into this. After the 'rpm' job is done we get a
zipfile with all the built RPMs inside. In the future we may want to run 'createrepo_c' on this dir as well, so we'll have a ready-to-be-used repository in that zip.
We already have code to do this in automation/common.sh - can we reuse it or extract it to new script?
- 'tests' are working too, but we have a couple of failures we'd need to look at. This job, unlike the others, runs in GitHub's Ubuntu VM inside which we use a privileged container for running the tests.
The failure seems to be missing TRAVIS_CI=1 environment variable. This should of course change to something moer generic like CONTAINER_CI=1 and the helper modules tests/testing.py should be updated.
Tests pass adding env: TRAVIS_CI=1 Ugly but works :-)
- didn't try 'tests-storage' yet
That's fine, the storage team will handle this.
Added storage tests here: https://github.com/nirs/vdsm/commit/0e5282945e4c86e65d5a1c12987e7534f350f2f2
- Not to waist precious free CI minutes and storage, we run linters first, tests after that, but only under the condition that the linters didn't fail, and finally we build RPMs, but this time under the condition that the tests didn't fail.
This is very bad idea. We want to run all the jobs in parallel so we can get quick feedback from all jobs at the same run. There is no real dependency between the jobs, and we don't waste resources by running them in parallel.
Here is a build with all jobs running in parallel, finish in 6.5 minutes: https://github.com/nirs/vdsm/actions/runs/1512754118 I think we can add install tests to the rpm job. This is at least 3 times faster than jenkins. We can start using github CI now.
On Sun, Nov 28, 2021 at 3:44 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Nov 28, 2021 at 12:41 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Fri, Nov 26, 2021 at 7:44 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi All,
I've been working on adding GitHub Actions-based CI to vdsm today. Feel free to check out the patches here:
https://gerrit.ovirt.org/q/topic:%22github-actions-ci%22+(status:open%20OR%2...)
Awesome!
Some comments: - the linters work fine already, we can start using them
- RPM building works too in general. I think the RPM versions are not right yet,
rpm versions are corrected by: - pushing current master to your fork - pushing --tags to your fork - using "check-depth: 0" option for actions/checkout@v2
so I'll look into this. After the 'rpm' job is done we get a
zipfile with all the built RPMs inside. In the future we may want to run 'createrepo_c' on this dir as well, so we'll have a ready-to-be-used repository in that zip.
We already have code to do this in automation/common.sh - can we reuse it or extract it to new script?
- 'tests' are working too, but we have a couple of failures we'd need to look at. This job, unlike the others, runs in GitHub's Ubuntu VM inside which we use a privileged container for running the tests.
The failure seems to be missing TRAVIS_CI=1 environment variable. This should of course change to something moer generic like CONTAINER_CI=1 and the helper modules tests/testing.py should be updated.
Tests pass adding
env: TRAVIS_CI=1
Ugly but works :-)
- didn't try 'tests-storage' yet
That's fine, the storage team will handle this.
Added storage tests here: https://github.com/nirs/vdsm/commit/0e5282945e4c86e65d5a1c12987e7534f350f2f2
- Not to waist precious free CI minutes and storage, we run linters first, tests after that, but only under the condition that the linters didn't fail, and finally we build RPMs, but this time under the condition that the tests didn't fail.
This is very bad idea. We want to run all the jobs in parallel so we can get quick feedback from all jobs at the same run. There is no real dependency between the jobs, and we don't waste resources by running them in parallel.
Here is a build with all jobs running in parallel, finish in 6.5 minutes: https://github.com/nirs/vdsm/actions/runs/1512754118
I think we can add install tests to the rpm job.
This is at least 3 times faster than jenkins.
We can start using github CI now.
Here is also (much simpler) CI pipeline for gitlab: https://gitlab.com/nirs/vdsm/-/commit/dfaaf8f7311501307e4269fc0b427417ff5264... It works, except 3 network tests that need privileged container: https://gitlab.com/nirs/vdsm/-/pipelines/417881870 Network short summary: = 1 failed, 310 passed, 32 skipped, 4 xfailed, 2 xpassed, 1 warning, 2 errors in 17.12s = The long test (tests-storage) took 6 minutes, a little faster than github. The lint job is slower (, since gitlab provides a VM with one core, but our longest tests (storage) are not cpu bound, so it does not matter. The rpm job creates rpms correctly without jumping through any hoops: https://gitlab.com/nirs/vdsm/-/jobs/1826841070 Looks like both github and gitlab gives us better CI with very little effort. Nir
participants (2)
-
Marcin Sobczyk -
Nir Soffer