--I33OQG5IFrqF2BWt Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi! e have been having an issue with gerrit patches being merged before jenkins ran any tests on them, to avoid it from happening again I propose creating a new gerrit flag (Tests) with the following specifics: +1 - Tests passed/overrided 0 - Tests pending -1 - Tests broken where +1 is required to submit, +1 is set by jenkins when passing the tests and -1 is set by jenkins in case it breaks any tests. The +1 flag can be set also by maintainers to allow overriding the process. That way all the tests will be blocked until someone (hopefully jenkins) adds the +1 flag, but if the maintainer wants to override the value, she just has to set that flag herself. What do you think? --=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --I33OQG5IFrqF2BWt Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUhsQoAAoJEEBxx+HSYmnDhEAH/jWxTaroG11HCl6Ke547iHXl JS8sr8OF2Vc/D4iNgkopb+ZAM67tsT2DP/q3BtBLz34LlWiIiTgNOmW9Gve+0osM kXzyeTc2ZcyjCF2nvCazZm1hC0WKtgERGfc6wmULyUBgYySF9X0mznNdoWRANCfu CKreZx9KqfiSjGWXsEfog5ETJSX+luolpHzYi0nd9ZLuHKm76Edqn9b6X+CVewS5 Ev0vGmt/qdUDwM3sSPmp4hFVyGBQ/1vvnFgqvSSCp1WwEuFsyM7ctMKtsE0IotrB Tvwc2B3TWTpbKVTsaaiHSAjkRjiqiR6j4h5sEKagBUHhATxdq+7Z3jYZHM0Fn0I= =ZGaS -----END PGP SIGNATURE----- --I33OQG5IFrqF2BWt--
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 10:43:04 AM Subject: [ovirt-devel] Creating a new gerrit flag
Hi!
e have been having an issue with gerrit patches being merged before jenkins ran any tests on them, to avoid it from happening again I propose creating a new gerrit flag (Tests) with the following specifics:
+1 - Tests passed/overrided 0 - Tests pending -1 - Tests broken
where +1 is required to submit, +1 is set by jenkins when passing the tests and -1 is set by jenkins in case it breaks any tests. The +1 flag can be set also by maintainers to allow overriding the process.
That way all the tests will be blocked until someone (hopefully jenkins) adds the +1 flag, but if the maintainer wants to override the value, she just has to set that flag herself.
What do you think?
Looks good, but there is a scenario which worries me a bit. It happened in the past times that an otherwise good and working patch failed the tests because, for example, pep8 or pyflakes became stricter about the code formatting. Can the maintainer override such a -1 from failed tests in that case? (probably yes, but worth asking) Thanks, -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
----- Original Message -----
From: "Francesco Romani" <fromani@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: infra@ovirt.org, devel@ovirt.org Sent: Tuesday, December 9, 2014 11:48:00 AM Subject: Re: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 10:43:04 AM Subject: [ovirt-devel] Creating a new gerrit flag
Hi!
e have been having an issue with gerrit patches being merged before jenkins ran any tests on them, to avoid it from happening again I propose creating a new gerrit flag (Tests) with the following specifics:
+1 - Tests passed/overrided 0 - Tests pending -1 - Tests broken
where +1 is required to submit, +1 is set by jenkins when passing the tests and -1 is set by jenkins in case it breaks any tests. The +1 flag can be set also by maintainers to allow overriding the process.
That way all the tests will be blocked until someone (hopefully jenkins) adds the +1 flag, but if the maintainer wants to override the value, she just has to set that flag herself.
What do you think?
Looks good, but there is a scenario which worries me a bit.
It happened in the past times that an otherwise good and working patch failed the tests because, for example, pep8 or pyflakes became stricter about the code formatting.
Can the maintainer override such a -1 from failed tests in that case? (probably yes, but worth asking)
If a test fails on this, it will fail on all patches that will follow as well, so I don't think it's a bad requirement that you need to fix the underlying issues for patches to be merged. Without running tests you don't have any good wya to make sure a patch is 'good'.
Thanks,
-- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
What happens when rebasing? We can't afford waiting for tests to run on each rebase... as we might end up rebasing forever. ----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 11:43:04 AM Subject: [ovirt-devel] Creating a new gerrit flag
Hi!
e have been having an issue with gerrit patches being merged before jenkins ran any tests on them, to avoid it from happening again I propose creating a new gerrit flag (Tests) with the following specifics:
+1 - Tests passed/overrided 0 - Tests pending -1 - Tests broken
where +1 is required to submit, +1 is set by jenkins when passing the tests and -1 is set by jenkins in case it breaks any tests. The +1 flag can be set also by maintainers to allow overriding the process.
That way all the tests will be blocked until someone (hopefully jenkins) adds the +1 flag, but if the maintainer wants to override the value, she just has to set that flag herself.
What do you think?
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
----- Original Message -----
From: "Oved Ourfali" <ovedo@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: infra@ovirt.org, devel@ovirt.org Sent: Tuesday, December 9, 2014 11:52:11 AM Subject: Re: [ovirt-devel] Creating a new gerrit flag
What happens when rebasing? We can't afford waiting for tests to run on each rebase... as we might end up rebasing forever.
why can't you rerun tests after rebase to run tests if the patch is critical and pending?
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 11:43:04 AM Subject: [ovirt-devel] Creating a new gerrit flag
Hi!
e have been having an issue with gerrit patches being merged before jenkins ran any tests on them, to avoid it from happening again I propose creating a new gerrit flag (Tests) with the following specifics:
+1 - Tests passed/overrided 0 - Tests pending -1 - Tests broken
where +1 is required to submit, +1 is set by jenkins when passing the tests and -1 is set by jenkins in case it breaks any tests. The +1 flag can be set also by maintainers to allow overriding the process.
That way all the tests will be blocked until someone (hopefully jenkins) adds the +1 flag, but if the maintainer wants to override the value, she just has to set that flag herself.
What do you think?
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
----- Original Message -----
From: "Yaniv Dary" <ydary@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 12:10:36 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
From: "Oved Ourfali" <ovedo@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: infra@ovirt.org, devel@ovirt.org Sent: Tuesday, December 9, 2014 11:52:11 AM Subject: Re: [ovirt-devel] Creating a new gerrit flag
What happens when rebasing? We can't afford waiting for tests to run on each rebase... as we might end up rebasing forever.
why can't you rerun tests after rebase to run tests if the patch is critical and pending?
I meant manually.
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 11:43:04 AM Subject: [ovirt-devel] Creating a new gerrit flag
Hi!
e have been having an issue with gerrit patches being merged before jenkins ran any tests on them, to avoid it from happening again I propose creating a new gerrit flag (Tests) with the following specifics:
+1 - Tests passed/overrided 0 - Tests pending -1 - Tests broken
where +1 is required to submit, +1 is set by jenkins when passing the tests and -1 is set by jenkins in case it breaks any tests. The +1 flag can be set also by maintainers to allow overriding the process.
That way all the tests will be blocked until someone (hopefully jenkins) adds the +1 flag, but if the maintainer wants to override the value, she just has to set that flag herself.
What do you think?
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
----- Original Message -----
From: "Yaniv Dary" <ydary@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: infra@ovirt.org, devel@ovirt.org Sent: Tuesday, December 9, 2014 12:11:31 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
From: "Yaniv Dary" <ydary@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 12:10:36 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
From: "Oved Ourfali" <ovedo@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: infra@ovirt.org, devel@ovirt.org Sent: Tuesday, December 9, 2014 11:52:11 AM Subject: Re: [ovirt-devel] Creating a new gerrit flag
What happens when rebasing? We can't afford waiting for tests to run on each rebase... as we might end up rebasing forever.
why can't you rerun tests after rebase to run tests if the patch is critical and pending?
I meant manually.
You can run. The issue is that while running them, someone else can merge something, and then you'll need to rebase again... and again... and again...
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 11:43:04 AM Subject: [ovirt-devel] Creating a new gerrit flag
Hi!
e have been having an issue with gerrit patches being merged before jenkins ran any tests on them, to avoid it from happening again I propose creating a new gerrit flag (Tests) with the following specifics:
+1 - Tests passed/overrided 0 - Tests pending -1 - Tests broken
where +1 is required to submit, +1 is set by jenkins when passing the tests and -1 is set by jenkins in case it breaks any tests. The +1 flag can be set also by maintainers to allow overriding the process.
That way all the tests will be blocked until someone (hopefully jenkins) adds the +1 flag, but if the maintainer wants to override the value, she just has to set that flag herself.
What do you think?
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--6c2NcOVqGQ03X4Wi Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 12/09, Oved Ourfali wrote:
What happens when rebasing? We can't afford waiting for tests to run on each rebase... as we might en= d up rebasing forever.
For now we will have to, all the code that is going to be merged must be tested as it is going to be merged, that means running the tests in the last rebase too. In the future there are plans on using a gating system like zuul, so zuul will be the one monitoring the tests and merging when passes, so you will just add the flag, and that will trigger the gate, that runs the tests and merged the patch. It's unlikely that you'll have to wait forever, but there's nothing avoiding you doing that (right now even). I'd like to put emphasis again on differentiating between tests that are fast, that should run on each patch and tests that are slow, that should run on each merge. That will improve the feedback times.
=20 ----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 11:43:04 AM Subject: [ovirt-devel] Creating a new gerrit flag =20 Hi! =20 e have been having an issue with gerrit patches being merged before jenkins ran any tests on them, to avoid it from happening again I propose creating a new gerrit flag (Tests) with the following specifics: =20 =20 +1 - Tests passed/overrided 0 - Tests pending -1 - Tests broken =20 where +1 is required to submit, +1 is set by jenkins when passing the tests and -1 is set by jenkins in case it breaks any tests. The +1 flag can be set also by maintainers to allow overriding the process. =20 That way all the tests will be blocked until someone (hopefully jenkins) adds the +1 flag, but if the maintainer wants to override the value, she just has to set that flag herself. =20 =20 What do you think? =20 =20 -- David Caro =20 Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D =20 Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 =20 _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --6c2NcOVqGQ03X4Wi Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUhssDAAoJEEBxx+HSYmnDlmAH/jTGLANAfsNTi8QkXiDHONor Ye6SDEmj4Wz77VvFEskba9GDvCjKYUs9yf1dCI+ETba4+ilvlu+M9foEe4x0bpK+ EeVrKUxHyAP+y0hKvUoK1IsNBboJtYLOQcnAjPKnTzgemzvNjhzG+GvUJCVbrmn0 o7Jd1WEE3uCnjEgOnP5V1M+7cP/w0w6pSNhoT0taAU+T3ftXLKBZWZHU5k/hLGwc qSzUYWJlYeW9JYDa4Z0vsaxFUHuppYzpZI37oo7AR6ZoBobs/P+Se7fXUPfscEX3 5cCsM6De/1qZkdfB53b93LYn7VKxGwFAtULFunw5tWshlNb0wtBrupFnkaxmHuA= =BxeC -----END PGP SIGNATURE----- --6c2NcOVqGQ03X4Wi--
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: infra@ovirt.org, devel@ovirt.org Sent: Tuesday, December 9, 2014 12:12:19 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/09, Oved Ourfali wrote:
What happens when rebasing? We can't afford waiting for tests to run on each rebase... as we might end up rebasing forever.
For now we will have to, all the code that is going to be merged must be tested as it is going to be merged, that means running the tests in the last rebase too.
In the future there are plans on using a gating system like zuul, so zuul will be the one monitoring the tests and merging when passes, so you will just add the flag, and that will trigger the gate, that runs the tests and merged the patch.
It's unlikely that you'll have to wait forever, but there's nothing avoiding you doing that (right now even).
I'd like to put emphasis again on differentiating between tests that are fast, that should run on each patch and tests that are slow, that should run on each merge. That will improve the feedback times.
+1 Tests are more critical than fast merges, the consequences of merging untest patches is worse.
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 11:43:04 AM Subject: [ovirt-devel] Creating a new gerrit flag
Hi!
e have been having an issue with gerrit patches being merged before jenkins ran any tests on them, to avoid it from happening again I propose creating a new gerrit flag (Tests) with the following specifics:
+1 - Tests passed/overrided 0 - Tests pending -1 - Tests broken
where +1 is required to submit, +1 is set by jenkins when passing the tests and -1 is set by jenkins in case it breaks any tests. The +1 flag can be set also by maintainers to allow overriding the process.
That way all the tests will be blocked until someone (hopefully jenkins) adds the +1 flag, but if the maintainer wants to override the value, she just has to set that flag herself.
What do you think?
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: infra@ovirt.org, devel@ovirt.org Sent: Tuesday, December 9, 2014 12:12:19 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/09, Oved Ourfali wrote:
What happens when rebasing? We can't afford waiting for tests to run on each rebase... as we might end up rebasing forever.
For now we will have to, all the code that is going to be merged must be tested as it is going to be merged, that means running the tests in the last rebase too.
In the future there are plans on using a gating system like zuul, so zuul will be the one monitoring the tests and merging when passes, so you will just add the flag, and that will trigger the gate, that runs the tests and merged the patch.
It's unlikely that you'll have to wait forever, but there's nothing avoiding you doing that (right now even).
I'd like to put emphasis again on differentiating between tests that are fast, that should run on each patch and tests that are slow, that should run on each merge. That will improve the feedback times.
So let's apply that in the future. For now the amount of merges done is enormous, and it will be impossible to get things merged on a reasonable time. Again, I'm not against testing, but it should be done the right way...
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 11:43:04 AM Subject: [ovirt-devel] Creating a new gerrit flag
Hi!
e have been having an issue with gerrit patches being merged before jenkins ran any tests on them, to avoid it from happening again I propose creating a new gerrit flag (Tests) with the following specifics:
+1 - Tests passed/overrided 0 - Tests pending -1 - Tests broken
where +1 is required to submit, +1 is set by jenkins when passing the tests and -1 is set by jenkins in case it breaks any tests. The +1 flag can be set also by maintainers to allow overriding the process.
That way all the tests will be blocked until someone (hopefully jenkins) adds the +1 flag, but if the maintainer wants to override the value, she just has to set that flag herself.
What do you think?
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
top posting: How about the following flow: 1. You push a patch to gerrit. 2. You need +1 on Testing in order to merge it. 3. You have +1/-1 on the Tests if finished successfully/failed 4. You find out you need to rebase. 5. The rebase copies the result of the Tests of the previous patch-set... if it was +1, it remains +1 and you can merge (assuming you have +2 on CR). If it was -1 then you need to wait for the CI to finish, and it might set it to +1. Does that make sense? ----- Original Message -----
From: "Oved Ourfali" <ovedo@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 1:13:57 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: infra@ovirt.org, devel@ovirt.org Sent: Tuesday, December 9, 2014 12:12:19 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/09, Oved Ourfali wrote:
What happens when rebasing? We can't afford waiting for tests to run on each rebase... as we might end up rebasing forever.
For now we will have to, all the code that is going to be merged must be tested as it is going to be merged, that means running the tests in the last rebase too.
In the future there are plans on using a gating system like zuul, so zuul will be the one monitoring the tests and merging when passes, so you will just add the flag, and that will trigger the gate, that runs the tests and merged the patch.
It's unlikely that you'll have to wait forever, but there's nothing avoiding you doing that (right now even).
I'd like to put emphasis again on differentiating between tests that are fast, that should run on each patch and tests that are slow, that should run on each merge. That will improve the feedback times.
So let's apply that in the future. For now the amount of merges done is enormous, and it will be impossible to get things merged on a reasonable time. Again, I'm not against testing, but it should be done the right way...
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 11:43:04 AM Subject: [ovirt-devel] Creating a new gerrit flag
Hi!
e have been having an issue with gerrit patches being merged before jenkins ran any tests on them, to avoid it from happening again I propose creating a new gerrit flag (Tests) with the following specifics:
+1 - Tests passed/overrided 0 - Tests pending -1 - Tests broken
where +1 is required to submit, +1 is set by jenkins when passing the tests and -1 is set by jenkins in case it breaks any tests. The +1 flag can be set also by maintainers to allow overriding the process.
That way all the tests will be blocked until someone (hopefully jenkins) adds the +1 flag, but if the maintainer wants to override the value, she just has to set that flag herself.
What do you think?
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
----- Original Message -----
From: "Oved Ourfali" <ovedo@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: infra@ovirt.org, devel@ovirt.org Sent: Tuesday, December 9, 2014 2:27:14 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
top posting: How about the following flow: 1. You push a patch to gerrit. 2. You need +1 on Testing in order to merge it. 3. You have +1/-1 on the Tests if finished successfully/failed 4. You find out you need to rebase. 5. The rebase copies the result of the Tests of the previous patch-set... if it was +1, it remains +1 and you can merge (assuming you have +2 on CR). If it was -1 then you need to wait for the CI to finish, and it might set it to +1.
Does that make sense?
Yes, but only if you used rebase button and automatic rebase worked. In case of merge conflict you will need to wait after rebase for tests.
----- Original Message -----
From: "Oved Ourfali" <ovedo@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 1:13:57 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: infra@ovirt.org, devel@ovirt.org Sent: Tuesday, December 9, 2014 12:12:19 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/09, Oved Ourfali wrote:
What happens when rebasing? We can't afford waiting for tests to run on each rebase... as we might end up rebasing forever.
For now we will have to, all the code that is going to be merged must be tested as it is going to be merged, that means running the tests in the last rebase too.
In the future there are plans on using a gating system like zuul, so zuul will be the one monitoring the tests and merging when passes, so you will just add the flag, and that will trigger the gate, that runs the tests and merged the patch.
It's unlikely that you'll have to wait forever, but there's nothing avoiding you doing that (right now even).
I'd like to put emphasis again on differentiating between tests that are fast, that should run on each patch and tests that are slow, that should run on each merge. That will improve the feedback times.
So let's apply that in the future. For now the amount of merges done is enormous, and it will be impossible to get things merged on a reasonable time. Again, I'm not against testing, but it should be done the right way...
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 11:43:04 AM Subject: [ovirt-devel] Creating a new gerrit flag
Hi!
e have been having an issue with gerrit patches being merged before jenkins ran any tests on them, to avoid it from happening again I propose creating a new gerrit flag (Tests) with the following specifics:
+1 - Tests passed/overrided 0 - Tests pending -1 - Tests broken
where +1 is required to submit, +1 is set by jenkins when passing the tests and -1 is set by jenkins in case it breaks any tests. The +1 flag can be set also by maintainers to allow overriding the process.
That way all the tests will be blocked until someone (hopefully jenkins) adds the +1 flag, but if the maintainer wants to override the value, she just has to set that flag herself.
What do you think?
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
----- Original Message -----
From: "Yaniv Dary" <ydary@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 2:29:38 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
From: "Oved Ourfali" <ovedo@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: infra@ovirt.org, devel@ovirt.org Sent: Tuesday, December 9, 2014 2:27:14 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
top posting: How about the following flow: 1. You push a patch to gerrit. 2. You need +1 on Testing in order to merge it. 3. You have +1/-1 on the Tests if finished successfully/failed 4. You find out you need to rebase. 5. The rebase copies the result of the Tests of the previous patch-set... if it was +1, it remains +1 and you can merge (assuming you have +2 on CR). If it was -1 then you need to wait for the CI to finish, and it might set it to +1.
Does that make sense?
Yes, but only if you used rebase button and automatic rebase worked. In case of merge conflict you will need to wait after rebase for tests.
In such cases I think the reviewer should compare and decide, rather than wait for tests to finish.
----- Original Message -----
From: "Oved Ourfali" <ovedo@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 1:13:57 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: infra@ovirt.org, devel@ovirt.org Sent: Tuesday, December 9, 2014 12:12:19 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/09, Oved Ourfali wrote:
What happens when rebasing? We can't afford waiting for tests to run on each rebase... as we might end up rebasing forever.
For now we will have to, all the code that is going to be merged must be tested as it is going to be merged, that means running the tests in the last rebase too.
In the future there are plans on using a gating system like zuul, so zuul will be the one monitoring the tests and merging when passes, so you will just add the flag, and that will trigger the gate, that runs the tests and merged the patch.
It's unlikely that you'll have to wait forever, but there's nothing avoiding you doing that (right now even).
I'd like to put emphasis again on differentiating between tests that are fast, that should run on each patch and tests that are slow, that should run on each merge. That will improve the feedback times.
So let's apply that in the future. For now the amount of merges done is enormous, and it will be impossible to get things merged on a reasonable time. Again, I'm not against testing, but it should be done the right way...
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 11:43:04 AM Subject: [ovirt-devel] Creating a new gerrit flag
Hi!
e have been having an issue with gerrit patches being merged before jenkins ran any tests on them, to avoid it from happening again I propose creating a new gerrit flag (Tests) with the following specifics:
+1 - Tests passed/overrided 0 - Tests pending -1 - Tests broken
where +1 is required to submit, +1 is set by jenkins when passing the tests and -1 is set by jenkins in case it breaks any tests. The +1 flag can be set also by maintainers to allow overriding the process.
That way all the tests will be blocked until someone (hopefully jenkins) adds the +1 flag, but if the maintainer wants to override the value, she just has to set that flag herself.
What do you think?
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--H8ygTp4AXg6deix2 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 12/09, Yaniv Dary wrote:
=20 =20 ----- Original Message -----
From: "Oved Ourfali" <ovedo@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: infra@ovirt.org, devel@ovirt.org Sent: Tuesday, December 9, 2014 2:27:14 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag =20 top posting: How about the following flow: 1. You push a patch to gerrit. 2. You need +1 on Testing in order to merge it. 3. You have +1/-1 on the Tests if finished successfully/failed 4. You find out you need to rebase. 5. The rebase copies the result of the Tests of the previous patch-set.= =2E. if it was +1, it remains +1 and you can merge (assuming you have +2 on CR)= =2E If it was -1 then you need to wait for the CI to finish, and it might set = it to +1. =20 Does that make sense? =20 Yes, but only if you used rebase button and automatic rebase worked. In case of merge conflict you will need to wait after rebase for tests.
Well, that's more or less what's happening today, we did set up the flag propagation on trivial rebase time ago. I'll have to check how to make jenkins ignore those trivial rebases only if they have a +1. So are you sure that having no merge conflict means that the rebase patch works as before? (I imagine for example that you could have two features that together might not work well, even if they do not touch the same code)
=20
=20 ----- Original Message -----
From: "Oved Ourfali" <ovedo@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 1:13:57 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag =20 =20 =20 ----- Original Message -----
What happens when rebasing? We can't afford waiting for tests to run on each rebase... as we = might end up rebasing forever. =20 For now we will have to, all the code that is going to be merged mu= st be tested as it is going to be merged, that means running the tests= in
From: "David Caro" <dcaroest@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: infra@ovirt.org, devel@ovirt.org Sent: Tuesday, December 9, 2014 12:12:19 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag =20 On 12/09, Oved Ourfali wrote: the last rebase too. =20 In the future there are plans on using a gating system like zuul, so zuul will be the one monitoring the tests and merging when passes, = so you will just add the flag, and that will trigger the gate, that ru= ns the tests and merged the patch. =20 It's unlikely that you'll have to wait forever, but there's nothing avoiding you doing that (right now even). =20 I'd like to put emphasis again on differentiating between tests that are fast, that should run on each patch and tests that are slow, th= at should run on each merge. That will improve the feedback times. =20 =20 So let's apply that in the future. For now the amount of merges done is enormous, and it will be impossi= ble to get things merged on a reasonable time. Again, I'm not against testing, but it should be done the right way... =20
=20 ----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 11:43:04 AM Subject: [ovirt-devel] Creating a new gerrit flag =20 Hi! =20 e have been having an issue with gerrit patches being merged be= fore jenkins ran any tests on them, to avoid it from happening again= I propose creating a new gerrit flag (Tests) with the following specifics: =20 =20 +1 - Tests passed/overrided 0 - Tests pending -1 - Tests broken =20 where +1 is required to submit, +1 is set by jenkins when passing the tests and -1 is set by jenkins in case it breaks any tests. The +1 flag can be set also by maintainers to allow over= riding the process. =20 That way all the tests will be blocked until someone (hopefully jenkins) adds the +1 flag, but if the maintainer wants to overr= ide the value, she just has to set that flag herself. =20 =20 What do you think? =20 =20 -- David Caro =20 Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D =20 Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 =20 _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel =20 -- David Caro =20 Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D =20 Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 =20
Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel =20
Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel =20
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --H8ygTp4AXg6deix2 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUhu3LAAoJEEBxx+HSYmnDdP0H/3RLp7CSIAP9HgffU/hBC/G2 wEZzBNIgaeeZ4wkjaPjN8tijl7iXX+/1f3CsIZNfrCEkl5b+gd9MZYsoczPGuXxX p8VuYM0Z/MgBNrz6b8p72wAGD3NuaK/UDnlbE7hsaYVb72t2SBBpYbFAvi6h/OWf GfteLxFI6NH3EF6j0u08wE2msM4270G6dh9D6h3nvUzyYAoNUaGFpZKlGGuKV9XC tcuH+i+GGyzg6QF8jBVgLDYKLi2ZYMhfKJpzTNXI8gkcAJp6HRgVi3TrIS5xNpp2 +f688J+oH4c+EZgV/DPu93r4f+OzHnhfK0BNjGX4G0HqYx8la4QN+f2GyaPdgKs= =Oqyk -----END PGP SIGNATURE----- --H8ygTp4AXg6deix2--
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Yaniv Dary" <ydary@redhat.com> Cc: "Oved Ourfali" <ovedo@redhat.com>, devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 2:40:43 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/09, Yaniv Dary wrote:
----- Original Message -----
From: "Oved Ourfali" <ovedo@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: infra@ovirt.org, devel@ovirt.org Sent: Tuesday, December 9, 2014 2:27:14 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
top posting: How about the following flow: 1. You push a patch to gerrit. 2. You need +1 on Testing in order to merge it. 3. You have +1/-1 on the Tests if finished successfully/failed 4. You find out you need to rebase. 5. The rebase copies the result of the Tests of the previous patch-set... if it was +1, it remains +1 and you can merge (assuming you have +2 on CR). If it was -1 then you need to wait for the CI to finish, and it might set it to +1.
Does that make sense?
Yes, but only if you used rebase button and automatic rebase worked. In case of merge conflict you will need to wait after rebase for tests.
Well, that's more or less what's happening today, we did set up the flag propagation on trivial rebase time ago. I'll have to check how to make jenkins ignore those trivial rebases only if they have a +1.
So are you sure that having no merge conflict means that the rebase patch works as before? (I imagine for example that you could have two features that together might not work well, even if they do not touch the same code)
You minimize the probability that something will get wrong. It isn't 100%. Using Zuul is the right way to go, but until you have that I think what I proposed will make it both easy to use and maintain, and both safe up to 95% or so.
----- Original Message -----
From: "Oved Ourfali" <ovedo@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: devel@ovirt.org, infra@ovirt.org Sent: Tuesday, December 9, 2014 1:13:57 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: infra@ovirt.org, devel@ovirt.org Sent: Tuesday, December 9, 2014 12:12:19 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/09, Oved Ourfali wrote:
What happens when rebasing? We can't afford waiting for tests to run on each rebase... as we might end up rebasing forever.
For now we will have to, all the code that is going to be merged must be tested as it is going to be merged, that means running the tests in the last rebase too.
In the future there are plans on using a gating system like zuul, so zuul will be the one monitoring the tests and merging when passes, so you will just add the flag, and that will trigger the gate, that runs the tests and merged the patch.
It's unlikely that you'll have to wait forever, but there's nothing avoiding you doing that (right now even).
I'd like to put emphasis again on differentiating between tests that are fast, that should run on each patch and tests that are slow, that should run on each merge. That will improve the feedback times.
So let's apply that in the future. For now the amount of merges done is enormous, and it will be impossible to get things merged on a reasonable time. Again, I'm not against testing, but it should be done the right way...
----- Original Message ----- > From: "David Caro" <dcaroest@redhat.com> > To: devel@ovirt.org, infra@ovirt.org > Sent: Tuesday, December 9, 2014 11:43:04 AM > Subject: [ovirt-devel] Creating a new gerrit flag > > Hi! > > e have been having an issue with gerrit patches being merged > before > jenkins ran any tests on them, to avoid it from happening again I > propose creating a new gerrit flag (Tests) with the following > specifics: > > > +1 - Tests passed/overrided > 0 - Tests pending > -1 - Tests broken > > where +1 is required to submit, +1 is set by jenkins when > passing the tests and -1 is set by jenkins in case it breaks any > tests. The +1 flag can be set also by maintainers to allow > overriding > the process. > > That way all the tests will be blocked until someone (hopefully > jenkins) adds the +1 flag, but if the maintainer wants to > override > the > value, she just has to set that flag herself. > > > What do you think? > > > -- > David Caro > > Red Hat S.L. > Continuous Integration Engineer - EMEA ENG Virtualization R&D > > Tel.: +420 532 294 605 > Email: dcaro@redhat.com > Web: www.redhat.com > RHT Global #: 82-62605 > > _______________________________________________ > Devel mailing list > Devel@ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On 09/12/14 13:47, Oved Ourfali wrote:
safe up to 95% or so.
You just made up that number. I don't really understand why you would want to downgrade your code quality by circumventing tests. Maybe someone can elaborate on this a bit? -- Mit freundlichen Grüßen / Regards Sven Kieske Systemadministrator Mittwald CM Service GmbH & Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
----- Original Message -----
From: "Sven Kieske" <s.kieske@mittwald.de> To: devel@ovirt.org Sent: Tuesday, December 9, 2014 3:21:43 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 09/12/14 13:47, Oved Ourfali wrote:
safe up to 95% or so.
You just made up that number. I don't really understand why you would want to downgrade your code quality by circumventing tests.
Maybe someone can elaborate on this a bit?
It doesn't downgrade the code quality. It is just a way to ensure developers can both merge changes, and do it as safely as possible without relying on post-submit tools. The number is indeed invented... as I don't have real statistics, but it comes to say that it would be safe most of the time. After the patch is merged, if CI will fail, it is the responsibility of the developer to check that out and fix that.
-- Mit freundlichen Grüßen / Regards
Sven Kieske
Systemadministrator Mittwald CM Service GmbH & Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--n/aVsWSeQ4JHkrmm Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 12/09, Oved Ourfali wrote:
=20 =20 ----- Original Message -----
From: "Sven Kieske" <s.kieske@mittwald.de> To: devel@ovirt.org Sent: Tuesday, December 9, 2014 3:21:43 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag =20 =20 =20 On 09/12/14 13:47, Oved Ourfali wrote:
safe up to 95% or so. =20 You just made up that number. I don't really understand why you would want to downgrade your code quality by circumventing tests. =20 Maybe someone can elaborate on this a bit? =20 =20 It doesn't downgrade the code quality. It is just a way to ensure developers can both merge changes, and do it a= s safely as possible without relying on post-submit tools. The number is indeed invented... as I don't have real statistics, but it = comes to say that it would be safe most of the time. After the patch is merged, if CI will fail, it is the responsibility of t= he developer to check that out and fix that.
This thread was started to avoid getting to that point, as getting a failed patch inside the code means breaking all the other tests that run on top of it and that blocks all the development, not only that specific patch.
=20
-- Mit freundlichen Gr=FC=DFen / Regards =20 Sven Kieske =20 Systemadministrator Mittwald CM Service GmbH & Co. KG K=F6nigsberger Stra=DFe 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Gesch=E4ftsf=FChrer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhau= sen Komplement=E4rin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeyn= hausen _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --n/aVsWSeQ4JHkrmm Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUhvvOAAoJEEBxx+HSYmnDWPAIAJyDu1Mc1ZDBWnCm0pk06/KB GgOWGpJTsNb3mQIt9PeATg8zlr0HIko+iF+I4RiKVzbFVp0ZkTvrUn/vK102x4MI CAGTu9twyWScQWEhgc8RqOOJpP8wcpkk/Bf5vXPZunACVWyzouzx0Bge2nJkaUSg ik8a+bax9MnaT45R6NmUmC3P8/Eaj2ktavR6ZQLsX+Nq7C5SUKAgb1hShV+ZIu89 HEEgfExXslserA8Wd1G9V5qxDHzz5P7MFrU1DK6bFzzoE4uyOYwd5SUoBHYHySdh qJ0np5o9EEwmffpgxvnIuT2J00M7bIBJzoQ9AYm6/hj1sfeuzIYap/wgS3buK6g= =+hXp -----END PGP SIGNATURE----- --n/aVsWSeQ4JHkrmm--
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: "Sven Kieske" <s.kieske@mittwald.de>, devel@ovirt.org Sent: Tuesday, December 9, 2014 3:40:30 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/09, Oved Ourfali wrote:
----- Original Message -----
From: "Sven Kieske" <s.kieske@mittwald.de> To: devel@ovirt.org Sent: Tuesday, December 9, 2014 3:21:43 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 09/12/14 13:47, Oved Ourfali wrote:
safe up to 95% or so.
You just made up that number. I don't really understand why you would want to downgrade your code quality by circumventing tests.
Maybe someone can elaborate on this a bit?
It doesn't downgrade the code quality. It is just a way to ensure developers can both merge changes, and do it as safely as possible without relying on post-submit tools. The number is indeed invented... as I don't have real statistics, but it comes to say that it would be safe most of the time. After the patch is merged, if CI will fail, it is the responsibility of the developer to check that out and fix that.
This thread was started to avoid getting to that point, as getting a failed patch inside the code means breaking all the other tests that run on top of it and that blocks all the development, not only that specific patch.
The issue that started the discussion was an issue in which there was a Tests "-1" flag, and it was ignored. My suggestion will enforce that it won't be ignored. In more rare cases, in which the rebase is the source of the tests issue, then you'll find about it later.
-- Mit freundlichen Grüßen / Regards
Sven Kieske
Systemadministrator Mittwald CM Service GmbH & Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
=20 =20 ----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: "Sven Kieske" <s.kieske@mittwald.de>, devel@ovirt.org Sent: Tuesday, December 9, 2014 3:40:30 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag =20 On 12/09, Oved Ourfali wrote:
=20 =20 ----- Original Message -----
From: "Sven Kieske" <s.kieske@mittwald.de> To: devel@ovirt.org Sent: Tuesday, December 9, 2014 3:21:43 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag =20 =20 =20 On 09/12/14 13:47, Oved Ourfali wrote:
safe up to 95% or so. =20 You just made up that number. I don't really understand why you would want to downgrade your code quality by circumventing tests. =20 Maybe someone can elaborate on this a bit? =20 =20 It doesn't downgrade the code quality. It is just a way to ensure developers can both merge changes, and do = it as safely as possible without relying on post-submit tools. The number is indeed invented... as I don't have real statistics, but= it comes to say that it would be safe most of the time. After the patch is merged, if CI will fail, it is the responsibility = of the developer to check that out and fix that. =20 This thread was started to avoid getting to that point, as getting a failed patch inside the code means breaking all the other tests that run on top of it and that blocks all the development, not only that specific patch. =20 =20 The issue that started the discussion was an issue in which there was a T= ests "-1" flag, and it was ignored. My suggestion will enforce that it won't be ignored. In more rare cases, in which the rebase is the source of the tests issue,=
--rf72Gf+bfLC8kxKs Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 12/09, Oved Ourfali wrote: then you'll find about it later. I started the discussion, and I started it because a developer complained about not being able to merge a patch because it was failing the tests due to an already merged patch that was making all the builds to fail. And was trying to get a solution to avoid getting to that point where a patch is merged while breaking the tests. So in summary, you are suggestion this: Creating a new flag 'tested' with values +1, 0 and -1 that only jenkins and managers can set Block form submitting any patches that have a -1 Carry the value of that flag to following patches only if the flag was -1 That way if any of the previous patches failed any test you won't be able to merge it unless a maintainer accepts or the tests pass. But if your previous patchset passed, you will be able to merge right away. Is that correct? Just for comparison, right now anyone with submit privileges can submit any patch rebasing and submitting before jenkins gives it's review. I don't see a real advantage, as right now only maintainers should be able to submit, so the people doing the submit already know that the tests are failing and ignoring it, your solution will not prevent them from keep ignoring them.
=20 =20
=20
-- Mit freundlichen Gr=FC=DFen / Regards =20 Sven Kieske =20 Systemadministrator Mittwald CM Service GmbH & Co. KG K=F6nigsberger Stra=DFe 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Gesch=E4ftsf=FChrer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplement=E4rin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel =20 -- David Caro =20 Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D =20 Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 =20
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --rf72Gf+bfLC8kxKs Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUhys0AAoJEEBxx+HSYmnDrusIAJFzntcs1DEL85J2XT+avKrb FG9/LjSx+ZnFCleG6KxGXOYB6dngFIan1obc57IinhCp80WJLP2BCInbSbqKW6F7 Y/SHQEi/QZ1gELi649v72+ShGXGRVSkiud50CSPCEkZh8T2F7v5rhB2kVoYT3h5f gC4wcAIfy0QarMIs/rj5J7aC3AMjX8hqF5XgoKhW3Z71E1UxOA3TqiewXknKSGtS wCVHc0plE4hemS7ohp1uHxtJeFOgYJ33CzuCMPmnIiAmEmRwMk2dBSLdnrNJ8k/K ZRq2M9eTqnlRr15kg4qfmXQM1XGLoZ5jqlI5acEPE+/SUIgFLU+SoJoQ2GJer3U= =dybC -----END PGP SIGNATURE----- --rf72Gf+bfLC8kxKs--
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: devel@ovirt.org Sent: Tuesday, December 9, 2014 7:02:44 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/09, Oved Ourfali wrote:
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: "Sven Kieske" <s.kieske@mittwald.de>, devel@ovirt.org Sent: Tuesday, December 9, 2014 3:40:30 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/09, Oved Ourfali wrote:
----- Original Message -----
From: "Sven Kieske" <s.kieske@mittwald.de> To: devel@ovirt.org Sent: Tuesday, December 9, 2014 3:21:43 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 09/12/14 13:47, Oved Ourfali wrote:
safe up to 95% or so.
You just made up that number. I don't really understand why you would want to downgrade your code quality by circumventing tests.
Maybe someone can elaborate on this a bit?
It doesn't downgrade the code quality. It is just a way to ensure developers can both merge changes, and do it as safely as possible without relying on post-submit tools. The number is indeed invented... as I don't have real statistics, but it comes to say that it would be safe most of the time. After the patch is merged, if CI will fail, it is the responsibility of the developer to check that out and fix that.
This thread was started to avoid getting to that point, as getting a failed patch inside the code means breaking all the other tests that run on top of it and that blocks all the development, not only that specific patch.
The issue that started the discussion was an issue in which there was a Tests "-1" flag, and it was ignored. My suggestion will enforce that it won't be ignored. In more rare cases, in which the rebase is the source of the tests issue, then you'll find about it later.
I started the discussion, and I started it because a developer complained about not being able to merge a patch because it was failing the tests due to an already merged patch that was making all the builds to fail. And was trying to get a solution to avoid getting to that point where a patch is merged while breaking the tests.
So in summary, you are suggestion this:
Creating a new flag 'tested' with values +1, 0 and -1 that only jenkins and managers can set
Block form submitting any patches that have a -1
Carry the value of that flag to following patches only if the flag was -1
And carry on also +1 in case of gerrit rebases.
That way if any of the previous patches failed any test you won't be able to merge it unless a maintainer accepts or the tests pass. But if your previous patchset passed, you will be able to merge right away.
Is that correct?
Just for comparison, right now anyone with submit privileges can submit any patch rebasing and submitting before jenkins gives it's review. I don't see a real advantage, as right now only maintainers should be able to submit, so the people doing the submit already know that the tests are failing and ignoring it, your solution will not prevent them from keep ignoring them.
-- Mit freundlichen Grüßen / Regards
Sven Kieske
Systemadministrator Mittwald CM Service GmbH & Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
----- Original Message -----
From: "Oved Ourfali" <ovedo@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: devel@ovirt.org Sent: Wednesday, December 10, 2014 8:30:30 AM Subject: Re: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: devel@ovirt.org Sent: Tuesday, December 9, 2014 7:02:44 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/09, Oved Ourfali wrote:
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: "Sven Kieske" <s.kieske@mittwald.de>, devel@ovirt.org Sent: Tuesday, December 9, 2014 3:40:30 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/09, Oved Ourfali wrote:
----- Original Message -----
From: "Sven Kieske" <s.kieske@mittwald.de> To: devel@ovirt.org Sent: Tuesday, December 9, 2014 3:21:43 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 09/12/14 13:47, Oved Ourfali wrote: > safe up to 95% or so.
You just made up that number. I don't really understand why you would want to downgrade your code quality by circumventing tests.
Maybe someone can elaborate on this a bit?
It doesn't downgrade the code quality. It is just a way to ensure developers can both merge changes, and do it as safely as possible without relying on post-submit tools. The number is indeed invented... as I don't have real statistics, but it comes to say that it would be safe most of the time. After the patch is merged, if CI will fail, it is the responsibility of the developer to check that out and fix that.
This thread was started to avoid getting to that point, as getting a failed patch inside the code means breaking all the other tests that run on top of it and that blocks all the development, not only that specific patch.
The issue that started the discussion was an issue in which there was a Tests "-1" flag, and it was ignored. My suggestion will enforce that it won't be ignored. In more rare cases, in which the rebase is the source of the tests issue, then you'll find about it later.
I started the discussion, and I started it because a developer complained about not being able to merge a patch because it was failing the tests due to an already merged patch that was making all the builds to fail. And was trying to get a solution to avoid getting to that point where a patch is merged while breaking the tests.
So in summary, you are suggestion this:
Creating a new flag 'tested' with values +1, 0 and -1 that only jenkins and managers can set
Block form submitting any patches that have a -1
Carry the value of that flag to following patches only if the flag was -1
+1, we need a way to block bad patches from being merged, even with a rebase in gerrit. going forward we're planning a few changes to the way jenkins jobs are run on ovirt ci, which will help reduce noise and imrove resources usages. 1. moving into a flow process, where critical jobs like unit tests/checkstyle will run first and only then other heavy jobs will run (integration/rpms/findbugs) 2. using gating system like zuul to do the merge after verification of sanity jobs automatically. these will take time of course and will be done gradually.
And carry on also +1 in case of gerrit rebases.
That way if any of the previous patches failed any test you won't be able to merge it unless a maintainer accepts or the tests pass. But if your previous patchset passed, you will be able to merge right away.
Is that correct?
Just for comparison, right now anyone with submit privileges can submit any patch rebasing and submitting before jenkins gives it's review. I don't see a real advantage, as right now only maintainers should be able to submit, so the people doing the submit already know that the tests are failing and ignoring it, your solution will not prevent them from keep ignoring them.
-- Mit freundlichen Grüßen / Regards
Sven Kieske
Systemadministrator Mittwald CM Service GmbH & Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: devel@ovirt.org Cc: "Oved Ourfali" <ovedo@redhat.com>, "infra" <infra@ovirt.org> Sent: Wednesday, December 10, 2014 10:40:47 AM Subject: Re: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
From: "Oved Ourfali" <ovedo@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: devel@ovirt.org Sent: Wednesday, December 10, 2014 8:30:30 AM Subject: Re: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: devel@ovirt.org Sent: Tuesday, December 9, 2014 7:02:44 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/09, Oved Ourfali wrote:
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: "Sven Kieske" <s.kieske@mittwald.de>, devel@ovirt.org Sent: Tuesday, December 9, 2014 3:40:30 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/09, Oved Ourfali wrote:
----- Original Message ----- > From: "Sven Kieske" <s.kieske@mittwald.de> > To: devel@ovirt.org > Sent: Tuesday, December 9, 2014 3:21:43 PM > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > > On 09/12/14 13:47, Oved Ourfali wrote: > > safe up to 95% or so. > > You just made up that number. > I don't really understand why you would want > to downgrade your code quality by circumventing tests. > > Maybe someone can elaborate on this a bit? >
It doesn't downgrade the code quality. It is just a way to ensure developers can both merge changes, and do it as safely as possible without relying on post-submit tools. The number is indeed invented... as I don't have real statistics, but it comes to say that it would be safe most of the time. After the patch is merged, if CI will fail, it is the responsibility of the developer to check that out and fix that.
This thread was started to avoid getting to that point, as getting a failed patch inside the code means breaking all the other tests that run on top of it and that blocks all the development, not only that specific patch.
The issue that started the discussion was an issue in which there was a Tests "-1" flag, and it was ignored. My suggestion will enforce that it won't be ignored. In more rare cases, in which the rebase is the source of the tests issue, then you'll find about it later.
I started the discussion, and I started it because a developer complained about not being able to merge a patch because it was failing the tests due to an already merged patch that was making all the builds to fail. And was trying to get a solution to avoid getting to that point where a patch is merged while breaking the tests.
So in summary, you are suggestion this:
Creating a new flag 'tested' with values +1, 0 and -1 that only jenkins and managers can set
Block form submitting any patches that have a -1
Carry the value of that flag to following patches only if the flag was -1
+1, we need a way to block bad patches from being merged, even with a rebase in gerrit. going forward we're planning a few changes to the way jenkins jobs are run on ovirt ci, which will help reduce noise and imrove resources usages.
1. moving into a flow process, where critical jobs like unit tests/checkstyle will run first and only then other heavy jobs will run (integration/rpms/findbugs)
This is already implemented in vdsm for few months - running "make check" will run the fast tests first and will not run the slower tests if a fast test failed. Nir
--AGZzQgpsuUlWC1xT Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 12/10, Nir Soffer wrote: >=20 >=20 > ----- Original Message ----- > > From: "Eyal Edri" <eedri@redhat.com> > > To: devel@ovirt.org > > Cc: "Oved Ourfali" <ovedo@redhat.com>, "infra" <infra@ovirt.org> > > Sent: Wednesday, December 10, 2014 10:40:47 AM > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > >=20 > >=20 > >=20 > > ----- Original Message ----- > > > From: "Oved Ourfali" <ovedo@redhat.com> > > > To: "David Caro" <dcaroest@redhat.com> > > > Cc: devel@ovirt.org > > > Sent: Wednesday, December 10, 2014 8:30:30 AM > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > >=20 > > >=20 > > >=20 > > > ----- Original Message ----- > > > > From: "David Caro" <dcaroest@redhat.com> > > > > To: "Oved Ourfali" <ovedo@redhat.com> > > > > Cc: devel@ovirt.org > > > > Sent: Tuesday, December 9, 2014 7:02:44 PM > > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > >=20 > > > > On 12/09, Oved Ourfali wrote: > > > > >=20 > > > > >=20 > > > > > ----- Original Message ----- > > > > > > From: "David Caro" <dcaroest@redhat.com> > > > > > > To: "Oved Ourfali" <ovedo@redhat.com> > > > > > > Cc: "Sven Kieske" <s.kieske@mittwald.de>, devel@ovirt.org > > > > > > Sent: Tuesday, December 9, 2014 3:40:30 PM > > > > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > > > >=20 > > > > > > On 12/09, Oved Ourfali wrote: > > > > > > >=20 > > > > > > >=20 > > > > > > > ----- Original Message ----- > > > > > > > > From: "Sven Kieske" <s.kieske@mittwald.de> > > > > > > > > To: devel@ovirt.org > > > > > > > > Sent: Tuesday, December 9, 2014 3:21:43 PM > > > > > > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > > > > > >=20 > > > > > > > >=20 > > > > > > > >=20 > > > > > > > > On 09/12/14 13:47, Oved Ourfali wrote: > > > > > > > > > safe up to 95% or so. > > > > > > > >=20 > > > > > > > > You just made up that number. > > > > > > > > I don't really understand why you would want > > > > > > > > to downgrade your code quality by circumventing tests. > > > > > > > >=20 > > > > > > > > Maybe someone can elaborate on this a bit? > > > > > > > >=20 > > > > > > >=20 > > > > > > > It doesn't downgrade the code quality. > > > > > > > It is just a way to ensure developers can both merge changes,= and > > > > > > > do > > > > > > > it > > > > > > > as > > > > > > > safely as possible without relying on post-submit tools. > > > > > > > The number is indeed invented... as I don't have real statist= ics, > > > > > > > but > > > > > > > it > > > > > > > comes to say that it would be safe most of the time. > > > > > > > After the patch is merged, if CI will fail, it is the > > > > > > > responsibility > > > > > > > of > > > > > > > the > > > > > > > developer to check that out and fix that. > > > > > >=20 > > > > > > This thread was started to avoid getting to that point, as gett= ing a > > > > > > failed patch inside the code means breaking all the other tests= that > > > > > > run on top of it and that blocks all the development, not only = that > > > > > > specific patch. > > > > > >=20 > > > > >=20 > > > > > The issue that started the discussion was an issue in which there= was a > > > > > Tests "-1" flag, and it was ignored. > > > > > My suggestion will enforce that it won't be ignored. > > > > > In more rare cases, in which the rebase is the source of the tests > > > > > issue, > > > > > then you'll find about it later. > > > >=20 > > > > I started the discussion, and I started it because a developer > > > > complained about not being able to merge a patch because it was > > > > failing the tests due to an already merged patch that was making all > > > > the builds to fail. And was trying to get a solution to avoid getti= ng > > > > to that point where a patch is merged while breaking the tests. > > > >=20 > > > >=20 > > > > So in summary, you are suggestion this: > > > >=20 > > > > Creating a new flag 'tested' with values +1, 0 and -1 that only jen= kins > > > > and managers can set > > > >=20 > > > > Block form submitting any patches that have a -1 > > > >=20 > > > > Carry the value of that flag to following patches only if the flag = was > > > > -1 > > > >=20 > > >=20 > >=20 > > +1, we need a way to block bad patches from being merged, even with a r= ebase > > in gerrit. > > going forward we're planning a few changes to the way jenkins jobs are = run on > > ovirt ci, which will help > > reduce noise and imrove resources usages. > >=20 > > 1. moving into a flow process, where critical jobs like unit tests/chec= kstyle > > will run first and only then other heavy jobs will run > > (integration/rpms/findbugs) >=20 > This is already implemented in vdsm for few months - running "make check" > will run the fast tests first and will not run the slower tests if a fast= test > failed. Please change to be able to run only fast tests or only slow tests, that way we can separate the job into two and give feedback about the fast tests before the slows have finished running. Actually what eyal is talking about is not inside the project flow, but jenkins build pipeline. Thar ranges from static checks, unit tests, functional tests, builds and deployments (in the future). So instead of having one job for each step and running all of them in parallel, you'll run in a hierarchical manner, to avoid having to wat for all the tests to get feedback or failing before starting the most complex long-running tests. >=20 > Nir > _______________________________________________ > Infra mailing list > Infra@ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra --=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --AGZzQgpsuUlWC1xT Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUiF/YAAoJEEBxx+HSYmnD7HcH+wRFNKrzqUS/JO7f5YQCZxu3 4U4D/gKyURIPI0ghdf8vmEWbBQq4wKanKaO533rPuu9ZhuJRNOLhHCBA+I7rVdvL eyGN4E0gidWDb5tBtp5PaoRauEs7tC5DGMH0ZzyskfrBZeyUUjtbbmsCnAM7noya gx4nQh7lbE0e28ZbOt2GonAPVVkoYvQ+msd8v9/pG7Hks+KkO+24BuTfPRecSl3c qslRw09jBJoTdIuO4irfVqtUO4engLckFu02Q3ID5Q+qsXscG+AGGSvSzOVLtr3V 1WXalQ1qNd15+MB0q6n4XgQamRzsANQ8glcuSIdWeR2b96xuwrjz1MZeiSwUXdE= =AbWb -----END PGP SIGNATURE----- --AGZzQgpsuUlWC1xT--
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "Oved Ourfali" <ovedo@redhat.com>, devel@ovirt.org, "infra" <infra@ovirt.org> Sent: Wednesday, December 10, 2014 4:59:36 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/10, Nir Soffer wrote:
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: devel@ovirt.org Cc: "Oved Ourfali" <ovedo@redhat.com>, "infra" <infra@ovirt.org> Sent: Wednesday, December 10, 2014 10:40:47 AM Subject: Re: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
From: "Oved Ourfali" <ovedo@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: devel@ovirt.org Sent: Wednesday, December 10, 2014 8:30:30 AM Subject: Re: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: devel@ovirt.org Sent: Tuesday, December 9, 2014 7:02:44 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/09, Oved Ourfali wrote:
----- Original Message ----- > From: "David Caro" <dcaroest@redhat.com> > To: "Oved Ourfali" <ovedo@redhat.com> > Cc: "Sven Kieske" <s.kieske@mittwald.de>, devel@ovirt.org > Sent: Tuesday, December 9, 2014 3:40:30 PM > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > On 12/09, Oved Ourfali wrote: > > > > > > ----- Original Message ----- > > > From: "Sven Kieske" <s.kieske@mittwald.de> > > > To: devel@ovirt.org > > > Sent: Tuesday, December 9, 2014 3:21:43 PM > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > > > > > > > > > > On 09/12/14 13:47, Oved Ourfali wrote: > > > > safe up to 95% or so. > > > > > > You just made up that number. > > > I don't really understand why you would want > > > to downgrade your code quality by circumventing tests. > > > > > > Maybe someone can elaborate on this a bit? > > > > > > > It doesn't downgrade the code quality. > > It is just a way to ensure developers can both merge changes, > > and > > do > > it > > as > > safely as possible without relying on post-submit tools. > > The number is indeed invented... as I don't have real > > statistics, > > but > > it > > comes to say that it would be safe most of the time. > > After the patch is merged, if CI will fail, it is the > > responsibility > > of > > the > > developer to check that out and fix that. > > This thread was started to avoid getting to that point, as > getting a > failed patch inside the code means breaking all the other tests > that > run on top of it and that blocks all the development, not only > that > specific patch. >
The issue that started the discussion was an issue in which there was a Tests "-1" flag, and it was ignored. My suggestion will enforce that it won't be ignored. In more rare cases, in which the rebase is the source of the tests issue, then you'll find about it later.
I started the discussion, and I started it because a developer complained about not being able to merge a patch because it was failing the tests due to an already merged patch that was making all the builds to fail. And was trying to get a solution to avoid getting to that point where a patch is merged while breaking the tests.
So in summary, you are suggestion this:
Creating a new flag 'tested' with values +1, 0 and -1 that only jenkins and managers can set
Block form submitting any patches that have a -1
Carry the value of that flag to following patches only if the flag was -1
+1, we need a way to block bad patches from being merged, even with a rebase in gerrit. going forward we're planning a few changes to the way jenkins jobs are run on ovirt ci, which will help reduce noise and imrove resources usages.
1. moving into a flow process, where critical jobs like unit tests/checkstyle will run first and only then other heavy jobs will run (integration/rpms/findbugs)
This is already implemented in vdsm for few months - running "make check" will run the fast tests first and will not run the slower tests if a fast test failed.
Please change to be able to run only fast tests or only slow tests, that way we can separate the job into two and give feedback about the fast tests before the slows have finished running.
Actually what eyal is talking about is not inside the project flow, but jenkins build pipeline. Thar ranges from static checks, unit tests, functional tests, builds and deployments (in the future).
So instead of having one job for each step and running all of them in parallel, you'll run in a hierarchical manner, to avoid having to wat for all the tests to get feedback or failing before starting the most complex long-running tests.
+1 for preventing failing test changes to be merged into master. I also agree that the biggest barrier for achieving that are the test that run ages and making them finish in a shorter time would enable us implement the proposed change. We could've achieve that by: 1. Splitting our repo into several smaller ones (according to the projects structure) so only the very relevant part of the project tests would be run on on every change merge. 2. Creating only fast running unit tests. 2.1. All test that run longer than 2 seconds should be reviewed and rewritten/eliminated. 2.2. Avoid creating the unit tests that check more than one class functionality. Using DI in our code should help us with that. 3. If we insist on long running tests, I guess they aren't proper unit tests, we should extract them into a separate test set and run them periodically (nightly).
Nir _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "Oved Ourfali" <ovedo@redhat.com>, "infra" <infra@ovirt.org>, devel@ovirt.org Sent: Wednesday, December 10, 2014 4:59:36 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/10, Nir Soffer wrote:
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: devel@ovirt.org Cc: "Oved Ourfali" <ovedo@redhat.com>, "infra" <infra@ovirt.org> Sent: Wednesday, December 10, 2014 10:40:47 AM Subject: Re: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
From: "Oved Ourfali" <ovedo@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: devel@ovirt.org Sent: Wednesday, December 10, 2014 8:30:30 AM Subject: Re: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: devel@ovirt.org Sent: Tuesday, December 9, 2014 7:02:44 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/09, Oved Ourfali wrote:
----- Original Message ----- > From: "David Caro" <dcaroest@redhat.com> > To: "Oved Ourfali" <ovedo@redhat.com> > Cc: "Sven Kieske" <s.kieske@mittwald.de>, devel@ovirt.org > Sent: Tuesday, December 9, 2014 3:40:30 PM > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > On 12/09, Oved Ourfali wrote: > > > > > > ----- Original Message ----- > > > From: "Sven Kieske" <s.kieske@mittwald.de> > > > To: devel@ovirt.org > > > Sent: Tuesday, December 9, 2014 3:21:43 PM > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > > > > > > > > > > On 09/12/14 13:47, Oved Ourfali wrote: > > > > safe up to 95% or so. > > > > > > You just made up that number. > > > I don't really understand why you would want > > > to downgrade your code quality by circumventing tests. > > > > > > Maybe someone can elaborate on this a bit? > > > > > > > It doesn't downgrade the code quality. > > It is just a way to ensure developers can both merge changes, > > and > > do > > it > > as > > safely as possible without relying on post-submit tools. > > The number is indeed invented... as I don't have real > > statistics, > > but > > it > > comes to say that it would be safe most of the time. > > After the patch is merged, if CI will fail, it is the > > responsibility > > of > > the > > developer to check that out and fix that. > > This thread was started to avoid getting to that point, as > getting a > failed patch inside the code means breaking all the other tests > that > run on top of it and that blocks all the development, not only > that > specific patch. >
The issue that started the discussion was an issue in which there was a Tests "-1" flag, and it was ignored. My suggestion will enforce that it won't be ignored. In more rare cases, in which the rebase is the source of the tests issue, then you'll find about it later.
I started the discussion, and I started it because a developer complained about not being able to merge a patch because it was failing the tests due to an already merged patch that was making all the builds to fail. And was trying to get a solution to avoid getting to that point where a patch is merged while breaking the tests.
So in summary, you are suggestion this:
Creating a new flag 'tested' with values +1, 0 and -1 that only jenkins and managers can set
Block form submitting any patches that have a -1
Carry the value of that flag to following patches only if the flag was -1
+1, we need a way to block bad patches from being merged, even with a rebase in gerrit. going forward we're planning a few changes to the way jenkins jobs are run on ovirt ci, which will help reduce noise and imrove resources usages.
1. moving into a flow process, where critical jobs like unit tests/checkstyle will run first and only then other heavy jobs will run (integration/rpms/findbugs)
This is already implemented in vdsm for few months - running "make check" will run the fast tests first and will not run the slower tests if a fast test failed.
Please change to be able to run only fast tests or only slow tests, that way we can separate the job into two and give feedback about the fast tests before the slows have finished running.
These are the available targets (from faster to slower): - gitignore - check that certain files are ignored - pyflakes - check common Python errors (e.g. unused imports) - pep8 - style check - check - run the fast checks above and if successful, the unittests Environment variables: NOSE_SLOW_TESTS=1 - enable slow tests (we have only few) NOSE_STRESS_TESTS=1 - enable stress tests (probably not useful for the CI) Note that the environment variables are used only for the tests in vdsm/tests there are few tests in various sub directories that do not use the test infrastructure in vdsm/tests. - check-all - run make check enbaling both slow and stress tests Do you need a separate target for the unittests? Nir
--qYrsQHciA3Wqs7Iv Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 12/11, Nir Soffer wrote: > ----- Original Message ----- > > From: "David Caro" <dcaroest@redhat.com> > > To: "Nir Soffer" <nsoffer@redhat.com> > > Cc: "Eyal Edri" <eedri@redhat.com>, "Oved Ourfali" <ovedo@redhat.com>, = "infra" <infra@ovirt.org>, devel@ovirt.org > > Sent: Wednesday, December 10, 2014 4:59:36 PM > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > >=20 > > On 12/10, Nir Soffer wrote: > > >=20 > > >=20 > > > ----- Original Message ----- > > > > From: "Eyal Edri" <eedri@redhat.com> > > > > To: devel@ovirt.org > > > > Cc: "Oved Ourfali" <ovedo@redhat.com>, "infra" <infra@ovirt.org> > > > > Sent: Wednesday, December 10, 2014 10:40:47 AM > > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > >=20 > > > >=20 > > > >=20 > > > > ----- Original Message ----- > > > > > From: "Oved Ourfali" <ovedo@redhat.com> > > > > > To: "David Caro" <dcaroest@redhat.com> > > > > > Cc: devel@ovirt.org > > > > > Sent: Wednesday, December 10, 2014 8:30:30 AM > > > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > > >=20 > > > > >=20 > > > > >=20 > > > > > ----- Original Message ----- > > > > > > From: "David Caro" <dcaroest@redhat.com> > > > > > > To: "Oved Ourfali" <ovedo@redhat.com> > > > > > > Cc: devel@ovirt.org > > > > > > Sent: Tuesday, December 9, 2014 7:02:44 PM > > > > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > > > >=20 > > > > > > On 12/09, Oved Ourfali wrote: > > > > > > >=20 > > > > > > >=20 > > > > > > > ----- Original Message ----- > > > > > > > > From: "David Caro" <dcaroest@redhat.com> > > > > > > > > To: "Oved Ourfali" <ovedo@redhat.com> > > > > > > > > Cc: "Sven Kieske" <s.kieske@mittwald.de>, devel@ovirt.org > > > > > > > > Sent: Tuesday, December 9, 2014 3:40:30 PM > > > > > > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > > > > > >=20 > > > > > > > > On 12/09, Oved Ourfali wrote: > > > > > > > > >=20 > > > > > > > > >=20 > > > > > > > > > ----- Original Message ----- > > > > > > > > > > From: "Sven Kieske" <s.kieske@mittwald.de> > > > > > > > > > > To: devel@ovirt.org > > > > > > > > > > Sent: Tuesday, December 9, 2014 3:21:43 PM > > > > > > > > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > > > > > > > >=20 > > > > > > > > > >=20 > > > > > > > > > >=20 > > > > > > > > > > On 09/12/14 13:47, Oved Ourfali wrote: > > > > > > > > > > > safe up to 95% or so. > > > > > > > > > >=20 > > > > > > > > > > You just made up that number. > > > > > > > > > > I don't really understand why you would want > > > > > > > > > > to downgrade your code quality by circumventing tests. > > > > > > > > > >=20 > > > > > > > > > > Maybe someone can elaborate on this a bit? > > > > > > > > > >=20 > > > > > > > > >=20 > > > > > > > > > It doesn't downgrade the code quality. > > > > > > > > > It is just a way to ensure developers can both merge chan= ges, > > > > > > > > > and > > > > > > > > > do > > > > > > > > > it > > > > > > > > > as > > > > > > > > > safely as possible without relying on post-submit tools. > > > > > > > > > The number is indeed invented... as I don't have real > > > > > > > > > statistics, > > > > > > > > > but > > > > > > > > > it > > > > > > > > > comes to say that it would be safe most of the time. > > > > > > > > > After the patch is merged, if CI will fail, it is the > > > > > > > > > responsibility > > > > > > > > > of > > > > > > > > > the > > > > > > > > > developer to check that out and fix that. > > > > > > > >=20 > > > > > > > > This thread was started to avoid getting to that point, as > > > > > > > > getting a > > > > > > > > failed patch inside the code means breaking all the other t= ests > > > > > > > > that > > > > > > > > run on top of it and that blocks all the development, not o= nly > > > > > > > > that > > > > > > > > specific patch. > > > > > > > >=20 > > > > > > >=20 > > > > > > > The issue that started the discussion was an issue in which t= here > > > > > > > was a > > > > > > > Tests "-1" flag, and it was ignored. > > > > > > > My suggestion will enforce that it won't be ignored. > > > > > > > In more rare cases, in which the rebase is the source of the = tests > > > > > > > issue, > > > > > > > then you'll find about it later. > > > > > >=20 > > > > > > I started the discussion, and I started it because a developer > > > > > > complained about not being able to merge a patch because it was > > > > > > failing the tests due to an already merged patch that was makin= g all > > > > > > the builds to fail. And was trying to get a solution to avoid g= etting > > > > > > to that point where a patch is merged while breaking the tests. > > > > > >=20 > > > > > >=20 > > > > > > So in summary, you are suggestion this: > > > > > >=20 > > > > > > Creating a new flag 'tested' with values +1, 0 and -1 that only > > > > > > jenkins > > > > > > and managers can set > > > > > >=20 > > > > > > Block form submitting any patches that have a -1 > > > > > >=20 > > > > > > Carry the value of that flag to following patches only if the f= lag > > > > > > was > > > > > > -1 > > > > > >=20 > > > > >=20 > > > >=20 > > > > +1, we need a way to block bad patches from being merged, even with= a > > > > rebase > > > > in gerrit. > > > > going forward we're planning a few changes to the way jenkins jobs = are > > > > run on > > > > ovirt ci, which will help > > > > reduce noise and imrove resources usages. > > > >=20 > > > > 1. moving into a flow process, where critical jobs like unit > > > > tests/checkstyle > > > > will run first and only then other heavy jobs will run > > > > (integration/rpms/findbugs) > > >=20 > > > This is already implemented in vdsm for few months - running "make ch= eck" > > > will run the fast tests first and will not run the slower tests if a = fast > > > test > > > failed. > >=20 > > Please change to be able to run only fast tests or only slow tests, > > that way we can separate the job into two and give feedback about the > > fast tests before the slows have finished running. >=20 > These are the available targets (from faster to slower): >=20 > - gitignore - check that certain files are ignored >=20 > - pyflakes - check common Python errors (e.g. unused imports) >=20 > - pep8 - style check >=20 > - check - run the fast checks above and if successful, the unittests >=20 > Environment variables: >=20 > NOSE_SLOW_TESTS=3D1 - enable slow tests (we have only few) > NOSE_STRESS_TESTS=3D1 - enable stress tests (probably not useful for th= e CI) >=20 > Note that the environment variables are used only for the tests in > vdsm/tests there are few tests in various sub directories that do=20 > not use the test infrastructure in vdsm/tests. >=20 > - check-all - run make check enbaling both slow and stress tests >=20 > Do you need a separate target for the unittests? I just want to be able to execute: > make fast-check for fast checks (you decide what are and what are not) > make slow-check for slow checks, non overlapping, meaning that if you want to runa ll the checks, you'll have to use fast and then slow. The idea is to generalize the interface we use for the tests between all the ovirt projects so from ci we don't have to go keeping specific scripts for each provect and for each project version. And be able to run the fast checks (on each patchset, merges included) and the slow ones (on each merge only) separated so we can give feedback faster and not start the slow ones. Then after that goes the building of the rpm, and then the functional tests. And last the release if relevant. On patch: fast_check On merge: fast_check -> slow_check -> build -> functional_check -> release That flow is generic for all the projects, so each project will have to implement the same interface to run each step (I don't really care if it's make fast-check or just a bash script that runs whatever you need underneath, the key is not having to specify any options if possible and running just one line, the same everywhere). That simplifies a LOT the maintenance of the jenkins jobs, and allows to change the way checks are run bound to the version of the product. Next step is to specify the required dependencies inside the project, so the dependencies for the tests are also stored in the repo and bound to the code, and installed in the test env on demand so no more issues with versioning or new dependencies. Right now that is hardcoded in the job itself, making it break if any future version needs a different dependency list. Not sure yet what's the best way to do that though, maybe a simple puppet manifest would be the way to go... because having a requirements.txt file is python specific and not all the projects we have are python or use distutils at all. Ideas are welcome. >=20 > Nir --=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --qYrsQHciA3Wqs7Iv Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUiWygAAoJEEBxx+HSYmnDaVEH/0zTqlMQzeJ0hlUJL17MO2GX eU5KaoxzaVW/GA3T7Vd4wikpuS4pj5Pwu+dg8sUK20T9W17fQeKBx9ODrXIJff84 yq3jPO7lzdPdt7GG+g4uHSygzym/5ABTaoaHWVm4u379q9PQNSxLVLl4mSudfw0L XBLLMyxZUGxiRvl2GESTjZtyD1U/sgr4ZpsZWTZKdUPF9kmK3/NUqCivaHx6whbH 6Yg2TtbhYL2tm2JXPdnQiCMg9D93B2VOrl5h47V3tTjbdYHYGjR6HC3KftSbmmQS 0YyA9KX2nzDN5mwwjErb8mvnUQz6OaFZD6jzKNrb88p0UPWMzD8WvoaQ4zX6E3g= =SWC6 -----END PGP SIGNATURE----- --qYrsQHciA3Wqs7Iv--
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "Oved Ourfali" <ovedo@redhat.com>, "infra" <infra@ovirt.org>, devel@ovirt.org Sent: Thursday, December 11, 2014 12:06:24 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/11, Nir Soffer wrote:
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "Oved Ourfali" <ovedo@redhat.com>, "infra" <infra@ovirt.org>, devel@ovirt.org Sent: Wednesday, December 10, 2014 4:59:36 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/10, Nir Soffer wrote:
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: devel@ovirt.org Cc: "Oved Ourfali" <ovedo@redhat.com>, "infra" <infra@ovirt.org> Sent: Wednesday, December 10, 2014 10:40:47 AM Subject: Re: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
From: "Oved Ourfali" <ovedo@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: devel@ovirt.org Sent: Wednesday, December 10, 2014 8:30:30 AM Subject: Re: [ovirt-devel] Creating a new gerrit flag
----- Original Message ----- > From: "David Caro" <dcaroest@redhat.com> > To: "Oved Ourfali" <ovedo@redhat.com> > Cc: devel@ovirt.org > Sent: Tuesday, December 9, 2014 7:02:44 PM > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > On 12/09, Oved Ourfali wrote: > > > > > > ----- Original Message ----- > > > From: "David Caro" <dcaroest@redhat.com> > > > To: "Oved Ourfali" <ovedo@redhat.com> > > > Cc: "Sven Kieske" <s.kieske@mittwald.de>, devel@ovirt.org > > > Sent: Tuesday, December 9, 2014 3:40:30 PM > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > > > > On 12/09, Oved Ourfali wrote: > > > > > > > > > > > > ----- Original Message ----- > > > > > From: "Sven Kieske" <s.kieske@mittwald.de> > > > > > To: devel@ovirt.org > > > > > Sent: Tuesday, December 9, 2014 3:21:43 PM > > > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > > > > > > > > > > > > > > > > > > On 09/12/14 13:47, Oved Ourfali wrote: > > > > > > safe up to 95% or so. > > > > > > > > > > You just made up that number. > > > > > I don't really understand why you would want > > > > > to downgrade your code quality by circumventing tests. > > > > > > > > > > Maybe someone can elaborate on this a bit? > > > > > > > > > > > > > It doesn't downgrade the code quality. > > > > It is just a way to ensure developers can both merge > > > > changes, > > > > and > > > > do > > > > it > > > > as > > > > safely as possible without relying on post-submit tools. > > > > The number is indeed invented... as I don't have real > > > > statistics, > > > > but > > > > it > > > > comes to say that it would be safe most of the time. > > > > After the patch is merged, if CI will fail, it is the > > > > responsibility > > > > of > > > > the > > > > developer to check that out and fix that. > > > > > > This thread was started to avoid getting to that point, as > > > getting a > > > failed patch inside the code means breaking all the other > > > tests > > > that > > > run on top of it and that blocks all the development, not > > > only > > > that > > > specific patch. > > > > > > > The issue that started the discussion was an issue in which > > there > > was a > > Tests "-1" flag, and it was ignored. > > My suggestion will enforce that it won't be ignored. > > In more rare cases, in which the rebase is the source of the > > tests > > issue, > > then you'll find about it later. > > I started the discussion, and I started it because a developer > complained about not being able to merge a patch because it was > failing the tests due to an already merged patch that was making > all > the builds to fail. And was trying to get a solution to avoid > getting > to that point where a patch is merged while breaking the tests. > > > So in summary, you are suggestion this: > > Creating a new flag 'tested' with values +1, 0 and -1 that only > jenkins > and managers can set > > Block form submitting any patches that have a -1 > > Carry the value of that flag to following patches only if the > flag > was > -1 >
+1, we need a way to block bad patches from being merged, even with a rebase in gerrit. going forward we're planning a few changes to the way jenkins jobs are run on ovirt ci, which will help reduce noise and imrove resources usages.
1. moving into a flow process, where critical jobs like unit tests/checkstyle will run first and only then other heavy jobs will run (integration/rpms/findbugs)
This is already implemented in vdsm for few months - running "make check" will run the fast tests first and will not run the slower tests if a fast test failed.
Please change to be able to run only fast tests or only slow tests, that way we can separate the job into two and give feedback about the fast tests before the slows have finished running.
These are the available targets (from faster to slower):
- gitignore - check that certain files are ignored
- pyflakes - check common Python errors (e.g. unused imports)
- pep8 - style check
- check - run the fast checks above and if successful, the unittests
Environment variables:
NOSE_SLOW_TESTS=1 - enable slow tests (we have only few) NOSE_STRESS_TESTS=1 - enable stress tests (probably not useful for the CI)
Note that the environment variables are used only for the tests in vdsm/tests there are few tests in various sub directories that do not use the test infrastructure in vdsm/tests.
- check-all - run make check enbaling both slow and stress tests
Do you need a separate target for the unittests?
I just want to be able to execute:
make fast-check
for fast checks (you decide what are and what are not)
make slow-check
for slow checks, non overlapping, meaning that if you want to runa ll the checks, you'll have to use fast and then slow.
The idea is to generalize the interface we use for the tests between all the ovirt projects so from ci we don't have to go keeping specific scripts for each provect and for each project version. And be able to run the fast checks (on each patchset, merges included) and the slow ones (on each merge only) separated so we can give feedback faster and not start the slow ones. Then after that goes the building of the rpm, and then the functional tests. And last the release if relevant.
On patch: fast_check
fast_check is not a good name - how about check-patch?
On merge: fast_check -> slow_check -> build -> functional_check -> release
Same: how about check-merge?
That flow is generic for all the projects, so each project will have to implement the same interface to run each step (I don't really care if it's make fast-check or just a bash script that runs whatever you need underneath, the key is not having to specify any options if possible and running just one line, the same everywhere).
These target do not make any sense out of the CI environment. I think you should implement these using a wrapper, adapting any project the the CI "interface". We can provide the building blocks using the makefile, so the CI can use them to build the required flow.
That simplifies a LOT the maintenance of the jenkins jobs, and allows to change the way checks are run bound to the version of the product.
Since each version may have different makefile, the CI wrapper must handle this. Nir
--xXygN3QAmJYWdGtb Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 12/11, Nir Soffer wrote: >=20 >=20 > ----- Original Message ----- > > From: "David Caro" <dcaroest@redhat.com> > > To: "Nir Soffer" <nsoffer@redhat.com> > > Cc: "Eyal Edri" <eedri@redhat.com>, "Oved Ourfali" <ovedo@redhat.com>, = "infra" <infra@ovirt.org>, devel@ovirt.org > > Sent: Thursday, December 11, 2014 12:06:24 PM > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > >=20 > > On 12/11, Nir Soffer wrote: > > > ----- Original Message ----- > > > > From: "David Caro" <dcaroest@redhat.com> > > > > To: "Nir Soffer" <nsoffer@redhat.com> > > > > Cc: "Eyal Edri" <eedri@redhat.com>, "Oved Ourfali" <ovedo@redhat.co= m>, > > > > "infra" <infra@ovirt.org>, devel@ovirt.org > > > > Sent: Wednesday, December 10, 2014 4:59:36 PM > > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > >=20 > > > > On 12/10, Nir Soffer wrote: > > > > >=20 > > > > >=20 > > > > > ----- Original Message ----- > > > > > > From: "Eyal Edri" <eedri@redhat.com> > > > > > > To: devel@ovirt.org > > > > > > Cc: "Oved Ourfali" <ovedo@redhat.com>, "infra" <infra@ovirt.org> > > > > > > Sent: Wednesday, December 10, 2014 10:40:47 AM > > > > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > > > >=20 > > > > > >=20 > > > > > >=20 > > > > > > ----- Original Message ----- > > > > > > > From: "Oved Ourfali" <ovedo@redhat.com> > > > > > > > To: "David Caro" <dcaroest@redhat.com> > > > > > > > Cc: devel@ovirt.org > > > > > > > Sent: Wednesday, December 10, 2014 8:30:30 AM > > > > > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > > > > >=20 > > > > > > >=20 > > > > > > >=20 > > > > > > > ----- Original Message ----- > > > > > > > > From: "David Caro" <dcaroest@redhat.com> > > > > > > > > To: "Oved Ourfali" <ovedo@redhat.com> > > > > > > > > Cc: devel@ovirt.org > > > > > > > > Sent: Tuesday, December 9, 2014 7:02:44 PM > > > > > > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > > > > > >=20 > > > > > > > > On 12/09, Oved Ourfali wrote: > > > > > > > > >=20 > > > > > > > > >=20 > > > > > > > > > ----- Original Message ----- > > > > > > > > > > From: "David Caro" <dcaroest@redhat.com> > > > > > > > > > > To: "Oved Ourfali" <ovedo@redhat.com> > > > > > > > > > > Cc: "Sven Kieske" <s.kieske@mittwald.de>, devel@ovirt.o= rg > > > > > > > > > > Sent: Tuesday, December 9, 2014 3:40:30 PM > > > > > > > > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > > > > > > > >=20 > > > > > > > > > > On 12/09, Oved Ourfali wrote: > > > > > > > > > > >=20 > > > > > > > > > > >=20 > > > > > > > > > > > ----- Original Message ----- > > > > > > > > > > > > From: "Sven Kieske" <s.kieske@mittwald.de> > > > > > > > > > > > > To: devel@ovirt.org > > > > > > > > > > > > Sent: Tuesday, December 9, 2014 3:21:43 PM > > > > > > > > > > > > Subject: Re: [ovirt-devel] Creating a new gerrit fl= ag > > > > > > > > > > > >=20 > > > > > > > > > > > >=20 > > > > > > > > > > > >=20 > > > > > > > > > > > > On 09/12/14 13:47, Oved Ourfali wrote: > > > > > > > > > > > > > safe up to 95% or so. > > > > > > > > > > > >=20 > > > > > > > > > > > > You just made up that number. > > > > > > > > > > > > I don't really understand why you would want > > > > > > > > > > > > to downgrade your code quality by circumventing tes= ts. > > > > > > > > > > > >=20 > > > > > > > > > > > > Maybe someone can elaborate on this a bit? > > > > > > > > > > > >=20 > > > > > > > > > > >=20 > > > > > > > > > > > It doesn't downgrade the code quality. > > > > > > > > > > > It is just a way to ensure developers can both merge > > > > > > > > > > > changes, > > > > > > > > > > > and > > > > > > > > > > > do > > > > > > > > > > > it > > > > > > > > > > > as > > > > > > > > > > > safely as possible without relying on post-submit too= ls. > > > > > > > > > > > The number is indeed invented... as I don't have real > > > > > > > > > > > statistics, > > > > > > > > > > > but > > > > > > > > > > > it > > > > > > > > > > > comes to say that it would be safe most of the time. > > > > > > > > > > > After the patch is merged, if CI will fail, it is the > > > > > > > > > > > responsibility > > > > > > > > > > > of > > > > > > > > > > > the > > > > > > > > > > > developer to check that out and fix that. > > > > > > > > > >=20 > > > > > > > > > > This thread was started to avoid getting to that point,= as > > > > > > > > > > getting a > > > > > > > > > > failed patch inside the code means breaking all the oth= er > > > > > > > > > > tests > > > > > > > > > > that > > > > > > > > > > run on top of it and that blocks all the development, n= ot > > > > > > > > > > only > > > > > > > > > > that > > > > > > > > > > specific patch. > > > > > > > > > >=20 > > > > > > > > >=20 > > > > > > > > > The issue that started the discussion was an issue in whi= ch > > > > > > > > > there > > > > > > > > > was a > > > > > > > > > Tests "-1" flag, and it was ignored. > > > > > > > > > My suggestion will enforce that it won't be ignored. > > > > > > > > > In more rare cases, in which the rebase is the source of = the > > > > > > > > > tests > > > > > > > > > issue, > > > > > > > > > then you'll find about it later. > > > > > > > >=20 > > > > > > > > I started the discussion, and I started it because a develo= per > > > > > > > > complained about not being able to merge a patch because it= was > > > > > > > > failing the tests due to an already merged patch that was m= aking > > > > > > > > all > > > > > > > > the builds to fail. And was trying to get a solution to avo= id > > > > > > > > getting > > > > > > > > to that point where a patch is merged while breaking the te= sts. > > > > > > > >=20 > > > > > > > >=20 > > > > > > > > So in summary, you are suggestion this: > > > > > > > >=20 > > > > > > > > Creating a new flag 'tested' with values +1, 0 and -1 that = only > > > > > > > > jenkins > > > > > > > > and managers can set > > > > > > > >=20 > > > > > > > > Block form submitting any patches that have a -1 > > > > > > > >=20 > > > > > > > > Carry the value of that flag to following patches only if t= he > > > > > > > > flag > > > > > > > > was > > > > > > > > -1 > > > > > > > >=20 > > > > > > >=20 > > > > > >=20 > > > > > > +1, we need a way to block bad patches from being merged, even = with a > > > > > > rebase > > > > > > in gerrit. > > > > > > going forward we're planning a few changes to the way jenkins j= obs > > > > > > are > > > > > > run on > > > > > > ovirt ci, which will help > > > > > > reduce noise and imrove resources usages. > > > > > >=20 > > > > > > 1. moving into a flow process, where critical jobs like unit > > > > > > tests/checkstyle > > > > > > will run first and only then other heavy jobs will run > > > > > > (integration/rpms/findbugs) > > > > >=20 > > > > > This is already implemented in vdsm for few months - running "make > > > > > check" > > > > > will run the fast tests first and will not run the slower tests i= f a > > > > > fast > > > > > test > > > > > failed. > > > >=20 > > > > Please change to be able to run only fast tests or only slow tests, > > > > that way we can separate the job into two and give feedback about t= he > > > > fast tests before the slows have finished running. > > >=20 > > > These are the available targets (from faster to slower): > > >=20 > > > - gitignore - check that certain files are ignored > > >=20 > > > - pyflakes - check common Python errors (e.g. unused imports) > > >=20 > > > - pep8 - style check > > >=20 > > > - check - run the fast checks above and if successful, the unittests > > >=20 > > > Environment variables: > > >=20 > > > NOSE_SLOW_TESTS=3D1 - enable slow tests (we have only few) > > > NOSE_STRESS_TESTS=3D1 - enable stress tests (probably not useful fo= r the > > > CI) > > >=20 > > > Note that the environment variables are used only for the tests in > > > vdsm/tests there are few tests in various sub directories that do > > > not use the test infrastructure in vdsm/tests. > > >=20 > > > - check-all - run make check enbaling both slow and stress tests > > >=20 > > > Do you need a separate target for the unittests? > >=20 > > I just want to be able to execute: > >=20 > > > make fast-check > >=20 > > for fast checks (you decide what are and what are not) > >=20 > > > make slow-check > >=20 > > for slow checks, non overlapping, meaning that if you want to runa ll > > the checks, you'll have to use fast and then slow. > >=20 > > The idea is to generalize the interface we use for the tests between > > all the ovirt projects so from ci we don't have to go keeping specific > > scripts for each provect and for each project version. And be able to > > run the fast checks (on each patchset, merges included) and the slow > > ones (on each merge only) separated so we can give feedback faster and > > not start the slow ones. Then after that goes the building of the rpm, > > and then the functional tests. And last the release if relevant. > >=20 > > On patch: > > fast_check >=20 > fast_check is not a good name - how about check-patch? That name is just a placeholder for the step where you run the fast checks, however you do it is not specified here. >=20 > >=20 > > On merge: > > fast_check -> slow_check -> build -> functional_check -> release >=20 > Same: how about check-merge? >=20 > >=20 > > That flow is generic for all the projects, so each project will have > > to implement the same interface to run each step (I don't really care > > if it's make fast-check or just a bash script that runs whatever you > > need underneath, the key is not having to specify any options if > > possible and running just one line, the same everywhere). >=20 > These target do not make any sense out of the CI environment. And that's part of the development process, as much as creating unit tests. I think that's the point that is creating all the fuss around this. For you CI is external to the development. >=20 > I think you should implement these using a wrapper, adapting any > project the the CI "interface". I totally disagree, because that 'adapter' is bound to a project and to a version, making it a perfect fit to have it inside the project code, changing it at the same time you change the code and adapting it to the code changes while they are made. Writing that adapter external to the project itself means that you have one adapter per project, and per project version, so doing some number, we have right now >50 projects so that would mean having to monitor all the 50 projects for each patch, and create a new adapet each time a change is made that modifies the way it runs the tests or the dependencies it needs. And bind that to that specific hash (and future ones). Compare that to having the 'adapter' inside the project, where you modify it on the same commit you change the way it builds or the dependencies, and just using that. If on the first step you need a team with 10% (random number) of the developers creating patches to keep up the pace, with the last you don't need anyone doing that job and you'll get no false failure and you'll have no 'adaptation' period where the code running does not have the correct deps or tests. >=20 > We can provide the building blocks using the makefile, so the CI > can use them to build the required flow. I don't really care what tools to use, I just want the interface implemented. If you decide to use a bash script ok for me, if it's a make targe, ok for ma, if its a perl script, ruby script, python script or anything else, ok for me too. Just the same everywhere. >=20 > >=20 > > That simplifies a LOT the maintenance of the jenkins jobs, and allows > > to change the way checks are run bound to the version of the product. >=20 > Since each version may have different makefile, the CI wrapper must > handle this. Totally disagree. Not implementing this means that the job will fail each time the code changes (because you can't maintain as many job versions as code changes in parallel and select each of them depending on the commit hash or whatever). This is a blocker to automate any CI and to keep a stable CI. If this is not done the stability will keep being low (because each dependency or version change will mean that the jobs fail, just like they are failing due to ioprocess or as they failed with pthread or pep8). And the maintenance will be very high, just like it is right now, requiring puppet and job changes each time a new dependency is needed and doing tricks and malabars when versions conflict, just like we do now, what enforces us to use specialized slaves and not being able to take advantage of all the resources we have. >=20 > Nir --=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --xXygN3QAmJYWdGtb Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUiX5ZAAoJEEBxx+HSYmnDJGgH/3gS5VE2DJeGDizi3Q9rjDn6 FA8hZRiAXYG+pqhxRZHaALcFWsFFr18wkD978QiubW1EQqE2AxIhgCKEk/QU2wF5 rySK42l40uoSOI7iMu1d0Ip4T+UjILX6qmhYI35DePKlJIUFMTyjrGja3jYAO0uT grQiXeLpUqKs/P1QYXGRCUfrskxy/hJi+m2xJWYl++AtLr+DDjF8z2dHn4y/AhCO KJJLdpUd2gQEFn2eMsPNqwe7P/PWfcaqed3vXVFR6gmK/ie6bY5iRfTX6yeK4nUB vWShzPSbN0et4AuUTli1TsjISitDmnRFGT26oCMWsJ3y+TYMU60NCKdPbU/xJms= =ep1l -----END PGP SIGNATURE----- --xXygN3QAmJYWdGtb--
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "Oved Ourfali" <ovedo@redhat.com>, "infra" <infra@ovirt.org>, devel@ovirt.org, "Dan Kenigsberg" <danken@redhat.com> Sent: Thursday, December 11, 2014 1:22:01 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/11, Nir Soffer wrote:
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "Oved Ourfali" <ovedo@redhat.com>, "infra" <infra@ovirt.org>, devel@ovirt.org Sent: Thursday, December 11, 2014 12:06:24 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/11, Nir Soffer wrote:
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "Oved Ourfali" <ovedo@redhat.com>, "infra" <infra@ovirt.org>, devel@ovirt.org Sent: Wednesday, December 10, 2014 4:59:36 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/10, Nir Soffer wrote:
----- Original Message ----- > From: "Eyal Edri" <eedri@redhat.com> > To: devel@ovirt.org > Cc: "Oved Ourfali" <ovedo@redhat.com>, "infra" <infra@ovirt.org> > Sent: Wednesday, December 10, 2014 10:40:47 AM > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > > ----- Original Message ----- > > From: "Oved Ourfali" <ovedo@redhat.com> > > To: "David Caro" <dcaroest@redhat.com> > > Cc: devel@ovirt.org > > Sent: Wednesday, December 10, 2014 8:30:30 AM > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > > > > > > ----- Original Message ----- > > > From: "David Caro" <dcaroest@redhat.com> > > > To: "Oved Ourfali" <ovedo@redhat.com> > > > Cc: devel@ovirt.org > > > Sent: Tuesday, December 9, 2014 7:02:44 PM > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > > > > On 12/09, Oved Ourfali wrote: > > > > > > > > > > > > ----- Original Message ----- > > > > > From: "David Caro" <dcaroest@redhat.com> > > > > > To: "Oved Ourfali" <ovedo@redhat.com> > > > > > Cc: "Sven Kieske" <s.kieske@mittwald.de>, devel@ovirt.org > > > > > Sent: Tuesday, December 9, 2014 3:40:30 PM > > > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > > > > > > > > On 12/09, Oved Ourfali wrote: > > > > > > > > > > > > > > > > > > ----- Original Message ----- > > > > > > > From: "Sven Kieske" <s.kieske@mittwald.de> > > > > > > > To: devel@ovirt.org > > > > > > > Sent: Tuesday, December 9, 2014 3:21:43 PM > > > > > > > Subject: Re: [ovirt-devel] Creating a new gerrit flag > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 09/12/14 13:47, Oved Ourfali wrote: > > > > > > > > safe up to 95% or so. > > > > > > > > > > > > > > You just made up that number. > > > > > > > I don't really understand why you would want > > > > > > > to downgrade your code quality by circumventing > > > > > > > tests. > > > > > > > > > > > > > > Maybe someone can elaborate on this a bit? > > > > > > > > > > > > > > > > > > > It doesn't downgrade the code quality. > > > > > > It is just a way to ensure developers can both merge > > > > > > changes, > > > > > > and > > > > > > do > > > > > > it > > > > > > as > > > > > > safely as possible without relying on post-submit > > > > > > tools. > > > > > > The number is indeed invented... as I don't have real > > > > > > statistics, > > > > > > but > > > > > > it > > > > > > comes to say that it would be safe most of the time. > > > > > > After the patch is merged, if CI will fail, it is the > > > > > > responsibility > > > > > > of > > > > > > the > > > > > > developer to check that out and fix that. > > > > > > > > > > This thread was started to avoid getting to that point, > > > > > as > > > > > getting a > > > > > failed patch inside the code means breaking all the other > > > > > tests > > > > > that > > > > > run on top of it and that blocks all the development, not > > > > > only > > > > > that > > > > > specific patch. > > > > > > > > > > > > > The issue that started the discussion was an issue in which > > > > there > > > > was a > > > > Tests "-1" flag, and it was ignored. > > > > My suggestion will enforce that it won't be ignored. > > > > In more rare cases, in which the rebase is the source of > > > > the > > > > tests > > > > issue, > > > > then you'll find about it later. > > > > > > I started the discussion, and I started it because a > > > developer > > > complained about not being able to merge a patch because it > > > was > > > failing the tests due to an already merged patch that was > > > making > > > all > > > the builds to fail. And was trying to get a solution to avoid > > > getting > > > to that point where a patch is merged while breaking the > > > tests. > > > > > > > > > So in summary, you are suggestion this: > > > > > > Creating a new flag 'tested' with values +1, 0 and -1 that > > > only > > > jenkins > > > and managers can set > > > > > > Block form submitting any patches that have a -1 > > > > > > Carry the value of that flag to following patches only if the > > > flag > > > was > > > -1 > > > > > > > +1, we need a way to block bad patches from being merged, even > with a > rebase > in gerrit. > going forward we're planning a few changes to the way jenkins > jobs > are > run on > ovirt ci, which will help > reduce noise and imrove resources usages. > > 1. moving into a flow process, where critical jobs like unit > tests/checkstyle > will run first and only then other heavy jobs will run > (integration/rpms/findbugs)
This is already implemented in vdsm for few months - running "make check" will run the fast tests first and will not run the slower tests if a fast test failed.
Please change to be able to run only fast tests or only slow tests, that way we can separate the job into two and give feedback about the fast tests before the slows have finished running.
These are the available targets (from faster to slower):
- gitignore - check that certain files are ignored
- pyflakes - check common Python errors (e.g. unused imports)
- pep8 - style check
- check - run the fast checks above and if successful, the unittests
Environment variables:
NOSE_SLOW_TESTS=1 - enable slow tests (we have only few) NOSE_STRESS_TESTS=1 - enable stress tests (probably not useful for the CI)
Note that the environment variables are used only for the tests in vdsm/tests there are few tests in various sub directories that do not use the test infrastructure in vdsm/tests.
- check-all - run make check enbaling both slow and stress tests
Do you need a separate target for the unittests?
I just want to be able to execute:
make fast-check
for fast checks (you decide what are and what are not)
make slow-check
for slow checks, non overlapping, meaning that if you want to runa ll the checks, you'll have to use fast and then slow.
The idea is to generalize the interface we use for the tests between all the ovirt projects so from ci we don't have to go keeping specific scripts for each provect and for each project version. And be able to run the fast checks (on each patchset, merges included) and the slow ones (on each merge only) separated so we can give feedback faster and not start the slow ones. Then after that goes the building of the rpm, and then the functional tests. And last the release if relevant.
On patch: fast_check
fast_check is not a good name - how about check-patch?
That name is just a placeholder for the step where you run the fast checks, however you do it is not specified here.
On merge: fast_check -> slow_check -> build -> functional_check -> release
Same: how about check-merge?
That flow is generic for all the projects, so each project will have to implement the same interface to run each step (I don't really care if it's make fast-check or just a bash script that runs whatever you need underneath, the key is not having to specify any options if possible and running just one line, the same everywhere).
These target do not make any sense out of the CI environment.
And that's part of the development process, as much as creating unit tests. I think that's the point that is creating all the fuss around this. For you CI is external to the development.
I think you should implement these using a wrapper, adapting any project the the CI "interface".
I totally disagree, because that 'adapter' is bound to a project and to a version, making it a perfect fit to have it inside the project code, changing it at the same time you change the code and adapting it to the code changes while they are made.
Writing that adapter external to the project itself means that you have one adapter per project, and per project version, so doing some number, we have right now >50 projects so that would mean having to monitor all the 50 projects for each patch, and create a new adapet each time a change is made that modifies the way it runs the tests or the dependencies it needs. And bind that to that specific hash (and future ones).
Compare that to having the 'adapter' inside the project, where you modify it on the same commit you change the way it builds or the dependencies, and just using that.
If on the first step you need a team with 10% (random number) of the developers creating patches to keep up the pace, with the last you don't need anyone doing that job and you'll get no false failure and you'll have no 'adaptation' period where the code running does not have the correct deps or tests.
Keeping the CI adapter in the project is fine, but it should be separate from the build system so we do not have to change the build system if we want to make changes in the CI.
We can provide the building blocks using the makefile, so the CI can use them to build the required flow.
I don't really care what tools to use, I just want the interface implemented. If you decide to use a bash script ok for me, if it's a make targe, ok for ma, if its a perl script, ruby script, python script or anything else, ok for me too. Just the same everywhere.
Ok, lets have these scripts in the root of each project: check-patch - run for each patch submitted check-merge - run before merging a patch So the CI can use the same code for any project.
That simplifies a LOT the maintenance of the jenkins jobs, and allows to change the way checks are run bound to the version of the product.
Since each version may have different makefile, the CI wrapper must handle this.
Totally disagree. Not implementing this means that the job will fail each time the code changes (because you can't maintain as many job versions as code changes in parallel and select each of them depending on the commit hash or whatever). This is a blocker to automate any CI and to keep a stable CI. If this is not done the stability will keep being low (because each dependency or version change will mean that the jobs fail, just like they are failing due to ioprocess or as they failed with pthread or pep8). And the maintenance will be very high, just like it is right now, requiring puppet and job changes each time a new dependency is needed and doing tricks and malabars when versions conflict, just like we do now, what enforces us to use specialized slaves and not being able to take advantage of all the resources we have.
We may need different scripts for different versions of a project, but that can be handled in the project by backporting the ci hooks. Nir
----- Original Message -----
From: "Oved Ourfali" <ovedo@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: devel@ovirt.org Sent: Tuesday, December 9, 2014 3:41:47 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Oved Ourfali" <ovedo@redhat.com> Cc: "Sven Kieske" <s.kieske@mittwald.de>, devel@ovirt.org Sent: Tuesday, December 9, 2014 3:40:30 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 12/09, Oved Ourfali wrote:
----- Original Message -----
From: "Sven Kieske" <s.kieske@mittwald.de> To: devel@ovirt.org Sent: Tuesday, December 9, 2014 3:21:43 PM Subject: Re: [ovirt-devel] Creating a new gerrit flag
On 09/12/14 13:47, Oved Ourfali wrote:
safe up to 95% or so.
You just made up that number. I don't really understand why you would want to downgrade your code quality by circumventing tests.
Maybe someone can elaborate on this a bit?
It doesn't downgrade the code quality. It is just a way to ensure developers can both merge changes, and do it as safely as possible without relying on post-submit tools. The number is indeed invented... as I don't have real statistics, but it comes to say that it would be safe most of the time. After the patch is merged, if CI will fail, it is the responsibility of the developer to check that out and fix that.
This thread was started to avoid getting to that point, as getting a failed patch inside the code means breaking all the other tests that run on top of it and that blocks all the development, not only that specific patch.
The issue that started the discussion was an issue in which there was a Tests "-1" flag, and it was ignored.
+1. Patches which fail jenkins jobs shouldn't be merged - this should be blocked imo. However, manual triggered jobs should get higher priority (if possible) on the jobs queue in order to overcome possible delays in merge of urgent patches.
My suggestion will enforce that it won't be ignored. In more rare cases, in which the rebase is the source of the tests issue, then you'll find about it later.
-- Mit freundlichen Grüßen / Regards
Sven Kieske
Systemadministrator Mittwald CM Service GmbH & Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
participants (9)
-
David Caro -
Eyal Edri -
Francesco Romani -
Moti Asayag -
Nir Soffer -
Oved Ourfali -
Sven Kieske -
Yaniv Dary -
Yevgeny Zaspitsky