4:06 a.m.
--qYrsQHciA3Wqs7Iv
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On 12/11, Nir Soffer wrote:
----- Original Message -----
> From: "David Caro" <dcaroest(a)redhat.com>
> To: "Nir Soffer" <nsoffer(a)redhat.com>
> Cc: "Eyal Edri" <eedri(a)redhat.com>, "Oved Ourfali"
<ovedo(a)redhat.com>, =
"infra" <infra(a)ovirt.org>,
devel(a)ovirt.org
> Sent: Wednesday, December 10, 2014 4:59:36 PM
> Subject: Re: [ovirt-devel] Creating a new gerrit flag
>=20
> On 12/10, Nir Soffer wrote:
> >=20
> >=20
> > ----- Original Message -----
> > > From: "Eyal Edri" <eedri(a)redhat.com>
> > > To: devel(a)ovirt.org
> > > Cc: "Oved Ourfali" <ovedo(a)redhat.com>, "infra"
<infra(a)ovirt.org>
> > > Sent: Wednesday, December 10, 2014 10:40:47 AM
> > > Subject: Re: [ovirt-devel] Creating a new gerrit flag
> > >=20
> > >=20
> > >=20
> > > ----- Original Message -----
> > > > From: "Oved Ourfali" <ovedo(a)redhat.com>
> > > > To: "David Caro" <dcaroest(a)redhat.com>
> > > > Cc: devel(a)ovirt.org
> > > > Sent: Wednesday, December 10, 2014 8:30:30 AM
> > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag
> > > >=20
> > > >=20
> > > >=20
> > > > ----- Original Message -----
> > > > > From: "David Caro" <dcaroest(a)redhat.com>
> > > > > To: "Oved Ourfali" <ovedo(a)redhat.com>
> > > > > Cc: devel(a)ovirt.org
> > > > > Sent: Tuesday, December 9, 2014 7:02:44 PM
> > > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag
> > > > >=20
> > > > > On 12/09, Oved Ourfali wrote:
> > > > > >=20
> > > > > >=20
> > > > > > ----- Original Message -----
> > > > > > > From: "David Caro"
<dcaroest(a)redhat.com>
> > > > > > > To: "Oved Ourfali" <ovedo(a)redhat.com>
> > > > > > > Cc: "Sven Kieske"
<s.kieske(a)mittwald.de>, devel(a)ovirt.org
> > > > > > > Sent: Tuesday, December 9, 2014 3:40:30 PM
> > > > > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag
> > > > > > >=20
> > > > > > > On 12/09, Oved Ourfali wrote:
> > > > > > > >=20
> > > > > > > >=20
> > > > > > > > ----- Original Message -----
> > > > > > > > > From: "Sven Kieske"
<s.kieske(a)mittwald.de>
> > > > > > > > > To: devel(a)ovirt.org
> > > > > > > > > Sent: Tuesday, December 9, 2014 3:21:43 PM
> > > > > > > > > Subject: Re: [ovirt-devel] Creating a new
gerrit flag
> > > > > > > > >=20
> > > > > > > > >=20
> > > > > > > > >=20
> > > > > > > > > On 09/12/14 13:47, Oved Ourfali wrote:
> > > > > > > > > > safe up to 95% or so.
> > > > > > > > >=20
> > > > > > > > > You just made up that number.
> > > > > > > > > I don't really understand why you would
want
> > > > > > > > > to downgrade your code quality by
circumventing tests.
> > > > > > > > >=20
> > > > > > > > > Maybe someone can elaborate on this a bit?
> > > > > > > > >=20
> > > > > > > >=20
> > > > > > > > It doesn't downgrade the code quality.
> > > > > > > > It is just a way to ensure developers can both
merge chan=
ges,
> > > > > > > > and
> > > > > > > > do
> > > > > > > > it
> > > > > > > > as
> > > > > > > > safely as possible without relying on post-submit
tools.
> > > > > > > > The number is indeed invented... as I don't
have real
> > > > > > > > statistics,
> > > > > > > > but
> > > > > > > > it
> > > > > > > > comes to say that it would be safe most of the
time.
> > > > > > > > After the patch is merged, if CI will fail, it is
the
> > > > > > > > responsibility
> > > > > > > > of
> > > > > > > > the
> > > > > > > > developer to check that out and fix that.
> > > > > > >=20
> > > > > > > This thread was started to avoid getting to that
point, as
> > > > > > > getting a
> > > > > > > failed patch inside the code means breaking all the
other t=
ests
> > > > > > > that
> > > > > > > run on top of it and that blocks all the development,
not o=
nly
> > > > > > > that
> > > > > > > specific patch.
> > > > > > >=20
> > > > > >=20
> > > > > > The issue that started the discussion was an issue in which
t=
here
> > > > > > was a
> > > > > > Tests "-1" flag, and it was ignored.
> > > > > > My suggestion will enforce that it won't be ignored.
> > > > > > In more rare cases, in which the rebase is the source of
the =
tests
> > > > > > issue,
> > > > > > then you'll find about it later.
> > > > >=20
> > > > > I started the discussion, and I started it because a developer
> > > > > complained about not being able to merge a patch because it was
> > > > > failing the tests due to an already merged patch that was makin=
g all
> > > > > the builds to fail. And was trying to get a
solution to avoid g=
etting
> > > > > to that point where a patch is merged while
breaking the tests.
> > > > >=20
> > > > >=20
> > > > > So in summary, you are suggestion this:
> > > > >=20
> > > > > Creating a new flag 'tested' with values +1, 0 and -1
that only
> > > > > jenkins
> > > > > and managers can set
> > > > >=20
> > > > > Block form submitting any patches that have a -1
> > > > >=20
> > > > > Carry the value of that flag to following patches only if the f=
lag
> > > > > was
> > > > > -1
> > > > >=20
> > > >=20
> > >=20
> > > +1, we need a way to block bad patches from being merged, even with=
a
> > > rebase
> > > in gerrit.
> > > going forward we're planning a few changes to the way jenkins jobs =
are
> > > run on
> > > ovirt ci, which will help
> > > reduce noise and imrove resources usages.
> > >=20
> > > 1. moving into a flow process, where critical jobs like unit
> > > tests/checkstyle
> > > will run first and only then other heavy jobs will run
> > > (integration/rpms/findbugs)
> >=20
> > This is already implemented in vdsm for few months - running "make ch=
eck"
> > will run the fast tests first and will not run the slower
tests if a =
fast
> > test
> > failed.
>=20
> Please change to be able to run only fast tests or only slow tests,
> that way we can separate the job into two and give feedback about the
> fast tests before the slows have finished running.
=20
These are the available targets (from faster to slower):
=20
- gitignore - check that certain files are ignored
=20
- pyflakes - check common Python errors (e.g. unused imports)
=20
- pep8 - style check
=20
- check - run the fast checks above and if successful, the unittests
=20
Environment variables:
=20
NOSE_SLOW_TESTS=3D1 - enable slow tests (we have only few)
NOSE_STRESS_TESTS=3D1 - enable stress tests (probably not useful for th=
e CI)
=20
Note that the environment variables are used only for the tests in
vdsm/tests there are few tests in various sub directories that do=20
not use the test infrastructure in vdsm/tests.
=20
- check-all - run make check enbaling both slow and stress tests
=20
Do you need a separate target for the unittests?
I just want to be able to execute:
make fast-check
for fast checks (you decide what are and what are not)
make slow-check
for slow checks, non overlapping, meaning that if you want to runa ll
the checks, you'll have to use fast and then slow.
The idea is to generalize the interface we use for the tests between
all the ovirt projects so from ci we don't have to go keeping specific
scripts for each provect and for each project version. And be able to
run the fast checks (on each patchset, merges included) and the slow
ones (on each merge only) separated so we can give feedback faster and
not start the slow ones. Then after that goes the building of the rpm,
and then the functional tests. And last the release if relevant.
On patch:
fast_check
On merge:
fast_check -> slow_check -> build -> functional_check -> release
That flow is generic for all the projects, so each project will have
to implement the same interface to run each step (I don't really care
if it's make fast-check or just a bash script that runs whatever you
need underneath, the key is not having to specify any options if
possible and running just one line, the same everywhere).
That simplifies a LOT the maintenance of the jenkins jobs, and allows
to change the way checks are run bound to the version of the product.
Next step is to specify the required dependencies inside the project,
so the dependencies for the tests are also stored in the repo and
bound to the code, and installed in the test env on demand so no more
issues with versioning or new dependencies. Right now that is
hardcoded in the job itself, making it break if any future version
needs a different dependency list. Not sure yet what's the best way to
do that though, maybe a simple puppet manifest would be the way to
go... because having a requirements.txt file is python specific and
not all the projects we have are python or use distutils at all.
Ideas are welcome.
=20
Nir
--=20
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605
Email: dcaro(a)redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
--qYrsQHciA3Wqs7Iv
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJUiWygAAoJEEBxx+HSYmnDaVEH/0zTqlMQzeJ0hlUJL17MO2GX
eU5KaoxzaVW/GA3T7Vd4wikpuS4pj5Pwu+dg8sUK20T9W17fQeKBx9ODrXIJff84
yq3jPO7lzdPdt7GG+g4uHSygzym/5ABTaoaHWVm4u379q9PQNSxLVLl4mSudfw0L
XBLLMyxZUGxiRvl2GESTjZtyD1U/sgr4ZpsZWTZKdUPF9kmK3/NUqCivaHx6whbH
6Yg2TtbhYL2tm2JXPdnQiCMg9D93B2VOrl5h47V3tTjbdYHYGjR6HC3KftSbmmQS
0YyA9KX2nzDN5mwwjErb8mvnUQz6OaFZD6jzKNrb88p0UPWMzD8WvoaQ4zX6E3g=
=SWC6
-----END PGP SIGNATURE-----
--qYrsQHciA3Wqs7Iv--