10:40 a.m.
----- Original Message -----
+1, we need a way to block bad patches from being merged, even with a rebase in gerrit.
going forward we're planning a few changes to the way jenkins jobs are run on ovirt
ci, which will help
reduce noise and imrove resources usages.
1. moving into a flow process, where critical jobs like unit tests/checkstyle will run
first and only then other heavy jobs will run (integration/rpms/findbugs)
2. using gating system like zuul to do the merge after verification of sanity jobs
automatically.
these will take time of course and will be done gradually.
4:59 p.m.
New subject: [ovirt-devel] Creating a new gerrit flag
--AGZzQgpsuUlWC1xT
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On 12/10, Nir Soffer wrote:
and
ics,
ing a
> > > > > failed patch inside the code means breaking
all the other tests=
that
> > > > > run on top of it and that blocks all the
development, not only =
that
was a
ng
kins
was
ebase
> in gerrit.
> going forward we're planning a few changes to the way jenkins jobs are =
run on
kstyle
test
failed.
Please change to be able to run only fast tests or only slow tests,
that way we can separate the job into two and give feedback about the
fast tests before the slows have finished running.
Actually what eyal is talking about is not inside the project flow,
but jenkins build pipeline. Thar ranges from static checks, unit
tests, functional tests, builds and deployments (in the future).
So instead of having one job for each step and running all of them in
parallel, you'll run in a hierarchical manner, to avoid having to wat
for all the tests to get feedback or failing before starting the most
complex long-running tests.
--=20
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605
Email: dcaro(a)redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
--AGZzQgpsuUlWC1xT
Content-Type: application/pgp-signature
--AGZzQgpsuUlWC1xT--
6:44 p.m.
New subject: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
+1 for preventing failing test changes to be merged into master.
I also agree that the biggest barrier for achieving that are the test that run ages
and making them finish in a shorter time would enable us implement the proposed change.
We could've achieve that by:
1. Splitting our repo into several smaller ones (according to the projects structure)
so only the very relevant part of the project tests would be run on on every change
merge.
2. Creating only fast running unit tests.
2.1. All test that run longer than 2 seconds should be reviewed and rewritten/eliminated.
2.2. Avoid creating the unit tests that check more than one class functionality.
Using DI in our code should help us with that.
3. If we insist on long running tests, I guess they aren't proper unit tests,
we should extract them into a separate test set and run them periodically (nightly).
11:53 a.m.
New subject: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
These are the available targets (from faster to slower):
- gitignore - check that certain files are ignored
- pyflakes - check common Python errors (e.g. unused imports)
- pep8 - style check
- check - run the fast checks above and if successful, the unittests
Environment variables:
NOSE_SLOW_TESTS=1 - enable slow tests (we have only few)
NOSE_STRESS_TESTS=1 - enable stress tests (probably not useful for the CI)
Note that the environment variables are used only for the tests in
vdsm/tests there are few tests in various sub directories that do
not use the test infrastructure in vdsm/tests.
- check-all - run make check enbaling both slow and stress tests
Do you need a separate target for the unittests?
Nir
12:06 p.m.
New subject: [ovirt-devel] Creating a new gerrit flag
--qYrsQHciA3Wqs7Iv
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On 12/11, Nir Soffer wrote:
"infra" <infra(a)ovirt.org>,
devel(a)ovirt.org
ges,
ests
> > > > > > > that
> > > > > > > run on top of it and that blocks all the development,
not o=
nly
here
tests
g all
> > > > > the builds to fail. And was trying to get a
solution to avoid g=
etting
lag
a
are
eck"
> > will run the fast tests first and will not run the slower
tests if a =
fast
e CI)
I just want to be able to execute:
make fast-check
for fast checks (you decide what are and what are not)
make slow-check
for slow checks, non overlapping, meaning that if you want to runa ll
the checks, you'll have to use fast and then slow.
The idea is to generalize the interface we use for the tests between
all the ovirt projects so from ci we don't have to go keeping specific
scripts for each provect and for each project version. And be able to
run the fast checks (on each patchset, merges included) and the slow
ones (on each merge only) separated so we can give feedback faster and
not start the slow ones. Then after that goes the building of the rpm,
and then the functional tests. And last the release if relevant.
On patch:
fast_check
On merge:
fast_check -> slow_check -> build -> functional_check -> release
That flow is generic for all the projects, so each project will have
to implement the same interface to run each step (I don't really care
if it's make fast-check or just a bash script that runs whatever you
need underneath, the key is not having to specify any options if
possible and running just one line, the same everywhere).
That simplifies a LOT the maintenance of the jenkins jobs, and allows
to change the way checks are run bound to the version of the product.
Next step is to specify the required dependencies inside the project,
so the dependencies for the tests are also stored in the repo and
bound to the code, and installed in the test env on demand so no more
issues with versioning or new dependencies. Right now that is
hardcoded in the job itself, making it break if any future version
needs a different dependency list. Not sure yet what's the best way to
do that though, maybe a simple puppet manifest would be the way to
go... because having a requirements.txt file is python specific and
not all the projects we have are python or use distutils at all.
Ideas are welcome.
=20
Nir
--=20
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605
Email: dcaro(a)redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
--qYrsQHciA3Wqs7Iv
Content-Type: application/pgp-signature
--qYrsQHciA3Wqs7Iv--
12:19 p.m.
New subject: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
fast_check is not a good name - how about check-patch?
On merge:
fast_check -> slow_check -> build -> functional_check -> release
Same: how about check-merge?
These target do not make any sense out of the CI environment.
I think you should implement these using a wrapper, adapting any
project the the CI "interface".
We can provide the building blocks using the makefile, so the CI
can use them to build the required flow.
That simplifies a LOT the maintenance of the jenkins jobs, and allows
to change the way checks are run bound to the version of the product.
Since each version may have different makefile, the CI wrapper must
handle this.
Nir
1:22 p.m.
New subject: [ovirt-devel] Creating a new gerrit flag
--xXygN3QAmJYWdGtb
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On 12/11, Nir Soffer wrote:
"infra" <infra(a)ovirt.org>,
devel(a)ovirt.org
m>,
rg
ag
ts.
ls.
as
> > > > > > > > > getting a
> > > > > > > > > failed patch inside the code means breaking
all the oth=
er
ot
ch
the
per
> > > > > > > complained about not being able to
merge a patch because it=
was
> > > > > > > failing the tests due to an
already merged patch that was m=
aking
> > > > > > > all
> > > > > > > the builds to fail. And was trying to get a solution
to avo=
id
> > > > > > > getting
> > > > > > > to that point where a patch is merged while breaking
the te=
sts.
only
he
with a
obs
f a
he
r
the
That name is just a placeholder for the step where you run the fast
checks, however you do it is not specified here.
And that's part of the development process, as much as creating unit
tests. I think that's the point that is creating all the fuss around
this. For you CI is external to the development.
=20
I think you should implement these using a wrapper, adapting any
project the the CI "interface".
I totally disagree, because that 'adapter' is bound to a project and
to a version, making it a perfect fit to have it inside the
project code, changing it at the same time you change the code and
adapting it to the code changes while they are made.
Writing that adapter external to the project itself means that you
have one adapter per project, and per project version, so doing some
number, we have right now >50 projects so that would mean having to
monitor all the 50 projects for each patch, and create a new adapet
each time a change is made that modifies the way it runs the tests or
the dependencies it needs. And bind that to that specific hash (and
future ones).
Compare that to having the 'adapter' inside the project, where you
modify it on the same commit you change the way it builds or the
dependencies, and just using that.
If on the first step you need a team with 10% (random number) of the
developers creating patches to keep up the pace, with the last you
don't need anyone doing that job and you'll get no false failure and
you'll have no 'adaptation' period where the code running does not
have the correct deps or tests.
=20
We can provide the building blocks using the makefile, so the CI
can use them to build the required flow.
I don't really care what tools to use, I just want the interface
implemented. If you decide to use a bash script ok for me, if it's a
make targe, ok for ma, if its a perl script, ruby script, python
script or anything else, ok for me too. Just the same everywhere.
Totally disagree. Not implementing this means that the job will fail
each time the code changes (because you can't maintain as many job
versions as code changes in parallel and select each of them depending
on the commit hash or whatever). This is a blocker to automate any
CI and to keep a stable CI.
If this is not done the stability will keep being low (because each
dependency or version change will mean that the jobs fail, just like
they are failing due to ioprocess or as they failed with pthread or
pep8).
And the maintenance will be very high, just like it is right now,
requiring puppet and job changes each time a new dependency is needed
and doing tricks and malabars when versions conflict, just like we do
now, what enforces us to use specialized slaves and not being able to
take advantage of all the resources we have.
=20
Nir
--=20
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605
Email: dcaro(a)redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
--xXygN3QAmJYWdGtb
Content-Type: application/pgp-signature
--xXygN3QAmJYWdGtb--
1:47 p.m.
New subject: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
Keeping the CI adapter in the project is fine, but it should be
separate from the build system so we do not have to change the
build system if we want to make changes in the CI.
Ok, lets have these scripts in the root of each project:
check-patch - run for each patch submitted
check-merge - run before merging a patch
So the CI can use the same code for any project.
We may need different scripts for different versions of a project,
but that can be handled in the project by backporting the ci
hooks.
Nir
3765
days inactive
3766
days old
8 comments
4 participants
participants (4)
-
David Caro -
Eyal Edri -
Nir Soffer -
Yevgeny Zaspitsky