[ATN] [ACTION REQUIRED] Tweaking engine CI flow
Hi all, Eyal and I sat together to analyse and tweak the engine CI and this is what we came up with: - dao test excludes updated - exclude dal/src/main/bundles from invoking dao tests A trivial update of validation messages without any db change would trigger dao test without any need. This is one less job run for lots of patches. This change is effective now. - Spare CI re-run on trivial rebases Gerrit trigger supports suppressing itself if the change to the tree was trivial. Most of the waste of resources(time and IO :) ) is around rebasing a change and waiting for CI to rerun. if Change1 is ci+1 and Change2 is ci+1 the chances that they will break CI together is very small and taking that risk is most probably worth it due to the huge resources waste This change isn't effective yet - *Please reply here* if you agree or not to make this change available. All of this is 'master' - 3.6 will follow if we will agree on activating that change. Thanks, Roy
On Wed, Mar 30, 2016 at 11:45 AM, Roy Golan <rgolan@redhat.com> wrote:
Hi all,
Eyal and I sat together to analyse and tweak the engine CI and this is what we came up with:
- dao test excludes updated - exclude dal/src/main/bundles from invoking dao tests A trivial update of validation messages without any db change would trigger dao test without any need. This is one less job run for lots of patches. This change is effective now.
- Spare CI re-run on trivial rebases Gerrit trigger supports suppressing itself if the change to the tree was trivial. Most of the waste of resources(time and IO :) ) is around rebasing a change and waiting for CI to rerun. if Change1 is ci+1 and Change2 is ci+1 the chances that they will break CI together is very small and taking that risk is most probably worth it due to the huge resources waste This change isn't effective yet - *Please reply here* if you agree or not to make this change available.
+1, I also think the gain here is huge both for CI resources and for developers having to rebase multiple times and wait more time to merge their patches. Even if there is a risk of 2 patches dependent and we'll fail on post merge once in a while, I think it outweight the alternative by far with benefits of resources & time. (If we'll see too many failures we can always revert the change).
All of this is 'master' - 3.6 will follow if we will agree on activating that change.
Thanks, Roy
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
--tpyx7gKuSYt+mjHM Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 03/30 11:45, Roy Golan wrote:
Hi all, =20 Eyal and I sat together to analyse and tweak the engine CI and this is wh= at we came up with: =20 - dao test excludes updated - exclude dal/src/main/bundles from invoking dao tests A trivial update of validation messages without any db change would trigger dao test without any need. This is one less job run for lots of patches. This change is effective now.
I really recommend merging this into the standard ci scripts and doing the trigger/don't trigger stuff there
=20 - Spare CI re-run on trivial rebases Gerrit trigger supports suppressing itself if the change to the tree was trivial. Most of the waste of resources(time and IO :) ) is around rebasing a change and waiting for CI to rerun. if Change1 is ci+1 and Change2 is ci+1 the chances that they will break CI together is very small and taking that risk is most probably worth it due to the huge resources waste
This is not exactly how it works, trivial rebases don't care if the previous patch had ci+1 or not, it just means that the rebase did not get any confli= cts, that most of the time is not related to the tests working or not (it might = be that someone change a method on another file, that your patch is using and = it will be a trivial rebase and break the tests/compile). I don't really recommend skipping them. What you might meant is non-code changes, those are changes that only change the commit message, for example= , if you have a patch, and you want to fix a typo in the commit message, current= ly that small commit message fix will require and trigger a ci run, while when skipping the non-code changes, it will not needed.
This change isn't effective yet - *Please reply here* if you agree or n= ot to make this change available. =20 All of this is 'master' - 3.6 will follow if we will agree on activating that change. =20 Thanks, Roy
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605 --tpyx7gKuSYt+mjHM Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJW+5TIAAoJEEBxx+HSYmnDbP8IAJkxIztpyvtCWlnQytV4HgQY x/Wq/K797i/8aMPSO9MKEc10QhYpdB0CoO8+n1Isak6LoE0G44+tFSnaWBHGaw0N 07FhPB+ye99eagiyUKPyENK8h5Xkth9xYDGJHIpLi1CReT4DQtIDGzGtCUQpSDuI KNzmpKdHDpoI5VxArHZVAynF7ifa6HvDCm8273cy1Ul5qmNFMaIi3D13OBaV6yRS VVpm5XMjq7OHbO8fWgwwU3hMDSTkWWxV0Y6G8ClQM9YtRFEtLeJg2whaMRIf1mgB vlLHut1aE0Hh/Gqr2taidhdscT8bgV1O8ZR2mXCG0zMpEwBWqVcXCkUlyfRE+Gk= =E2ZD -----END PGP SIGNATURE----- --tpyx7gKuSYt+mjHM--
On Wed, Mar 30, 2016 at 11:56 AM, David Caro <dcaro@redhat.com> wrote:
On 03/30 11:45, Roy Golan wrote:
Hi all,
Eyal and I sat together to analyse and tweak the engine CI and this is what we came up with:
- dao test excludes updated - exclude dal/src/main/bundles from invoking dao tests A trivial update of validation messages without any db change would trigger dao test without any need. This is one less job run for lots of patches. This change is effective now.
I really recommend merging this into the standard ci scripts and doing the trigger/don't trigger stuff there
Yea, thats the plan (discussed on another thread) but since it requires some logic added of excluding certain path we'll need to think how best to write it in check-patch.sh.
- Spare CI re-run on trivial rebases Gerrit trigger supports suppressing itself if the change to the tree
was
trivial. Most of the waste of resources(time and IO :) ) is around rebasing a change and waiting for CI to rerun. if Change1 is ci+1 and Change2 is ci+1 the chances that they will break CI together is very small and taking that risk is most probably worth it due to the huge resources waste
This is not exactly how it works, trivial rebases don't care if the previous patch had ci+1 or not, it just means that the rebase did not get any conflicts, that most of the time is not related to the tests working or not (it might be that someone change a method on another file, that your patch is using and it will be a trivial rebase and break the tests/compile).
Even so, I think its worth doing a test for a limited amount of time (1 week?) while everyone is aware of it and to see if it cause errors, the chances of 2 people working on the same file exactly during the same time and one change failing another is slim (If someone thinks otherwise and can give an example I'd be happy to hear), But since this is a gray area which we don't really know, I think its worth the test.
I don't really recommend skipping them. What you might meant is non-code changes, those are changes that only change the commit message, for example, if you have a patch, and you want to fix a typo in the commit message, currently that small commit message fix will require and trigger a ci run, while when skipping the non-code changes, it will not needed.
This change isn't effective yet - *Please reply here* if you agree or not to make this change available.
All of this is 'master' - 3.6 will follow if we will agree on activating that change.
Thanks, Roy
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
On Wed, Mar 30, 2016 at 11:56 AM, David Caro <dcaro@redhat.com> wrote: =20
On 03/30 11:45, Roy Golan wrote:
Hi all,
Eyal and I sat together to analyse and tweak the engine CI and this is what we came up with:
- dao test excludes updated - exclude dal/src/main/bundles from invok= ing dao tests A trivial update of validation messages without any db change would trigger dao test without any need. This is one less job run for lots = of patches. This change is effective now.
I really recommend merging this into the standard ci scripts and doing =
--V7BlxAaPrdhzdIM1 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 03/30 12:13, Eyal Edri wrote: the
trigger/don't trigger stuff there
=20 Yea, thats the plan (discussed on another thread) but since it requires some logic added of excluding certain path we'll need to think how best to write it in check-patch.sh.
=20 =20
- Spare CI re-run on trivial rebases Gerrit trigger supports suppressing itself if the change to the tree
was
trivial. Most of the waste of resources(time and IO :) ) is around rebasing a change and waiting for CI to rerun. if Change1 is ci+1 and Change2 is ci+1 the chances that they will break CI together is very small and taking that risk is most probably worth it due to the huge resour=
ces
waste
This is not exactly how it works, trivial rebases don't care if the previous patch had ci+1 or not, it just means that the rebase did not get any conflicts, that most of the time is not related to the tests working or not (it mi= ght be that someone change a method on another file, that your patch is using = and it will be a trivial rebase and break the tests/compile).
=20 Even so, I think its worth doing a test for a limited amount of time (1 week?) while everyone is aware of it and to see if it cause errors, the chances of 2 people working on the same file exactly during the same time and one change failing another is slim (If someone thinks otherwise and can give an example I'd be happy to hear), But since this is a gray area which we don't really know, I think its wor=
vdsm is already doing something similar, you can look to their scripts th
the test.
Imo the best approach here right now, in non-automatic tests, that is, run = the tests on demand, that will avoid running on unneeded patches, on rebases or anything but still require the tests to be run before merging. That's what the workflow flag was meant to solve (as it's a lot easier than adding a new gerrit plugin that adds a button or similar to the ui).
=20 =20
I don't really recommend skipping them. What you might meant is non-code changes, those are changes that only change the commit message, for example, if you have a patch, and you want to fix a typo in the commit message, currently that small commit message fix will require and trigger a ci run, while =
when
skipping the non-code changes, it will not needed.
This change isn't effective yet - *Please reply here* if you agree = or not to make this change available.
All of this is 'master' - 3.6 will follow if we will agree on activat= ing that change.
Thanks, Roy
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
=20 =20 =20 --=20 Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel =20 phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605 --V7BlxAaPrdhzdIM1 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJW+5xPAAoJEEBxx+HSYmnD1RMH/0aocBvcqt1eGnFnPTYieAyU XW7xjtmyADOxjhqkHWhbDWzEdV6AJs0mYEnsamUI8oUF6DYTIVJ2fZI6BOAxVR0W DbahzS5j5eXcRVK8ZKquhLShgFJSNuaiGvriUhhNapaEdEW73MdPZ2pn1GTTCyfQ sfT3KyGIm6gMs4KqIjVIK6Bt04X90ZpN16firUeU+yO25/sNcXevRv2dB7QPuWyf YFU/Z3uQGyC4q38bIFn4Rfryi5hgdB6nl7xWBXMTfNp5lHn9sGo4oo5Guvrd8JpH b713BTNPHZsqdLYefRBnFXiq+GntysTUvo5lGudnbaihvWENTetAPUHmGEXYRb8= =A4IX -----END PGP SIGNATURE----- --V7BlxAaPrdhzdIM1--
On Wed, Mar 30, 2016 at 11:56 AM, David Caro <dcaro@redhat.com> wrote:
On 03/30 11:45, Roy Golan wrote:
Hi all,
Eyal and I sat together to analyse and tweak the engine CI and this is what we came up with:
- dao test excludes updated - exclude dal/src/main/bundles from invoking dao tests A trivial update of validation messages without any db change would trigger dao test without any need. This is one less job run for lots of patches. This change is effective now.
I really recommend merging this into the standard ci scripts and doing the trigger/don't trigger stuff there
- Spare CI re-run on trivial rebases Gerrit trigger supports suppressing itself if the change to the tree
was
trivial. Most of the waste of resources(time and IO :) ) is around rebasing a change and waiting for CI to rerun. if Change1 is ci+1 and Change2 is ci+1 the chances that they will break CI together is very small and taking that risk is most probably worth it due to the huge resources waste
This is not exactly how it works, trivial rebases don't care if the previous patch had ci+1 or not, it just means that the rebase did not get any conflicts,
that most of the time is not related to the tests working or not (it might
be that someone change a method on another file, that your patch is using and it will be a trivial rebase and break the tests/compile).
Yes but this is very unlikely to happen. But there are obvious cases where it can:
If HEAD chaged a configuration file like pom files or other xml like
fixtures.xml so we can force a trigger if there was a diff in the rebase change list.
I don't really recommend skipping them. What you might meant is non-code changes, those are changes that only change the commit message, for example, if you have a patch, and you want to fix a typo in the commit message, currently that small commit message fix will require and trigger a ci run, while when skipping the non-code changes, it will not needed.
We should exclude non-code changes only too.
This change isn't effective yet - *Please reply here* if you agree or not
to make this change available.
All of this is 'master' - 3.6 will follow if we will agree on activating that change.
Thanks, Roy
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
On Wed, Mar 30, 2016 at 11:45 AM, Roy Golan <rgolan@redhat.com> wrote:
Hi all,
Eyal and I sat together to analyse and tweak the engine CI and this is what we came up with:
- dao test excludes updated - exclude dal/src/main/bundles from invoking dao tests A trivial update of validation messages without any db change would trigger dao test without any need. This is one less job run for lots of patches. This change is effective now.
- Spare CI re-run on trivial rebases Gerrit trigger supports suppressing itself if the change to the tree was trivial. Most of the waste of resources(time and IO :) ) is around rebasing a change and waiting for CI to rerun. if Change1 is ci+1 and Change2 is ci+1 the chances that they will break CI together is very small and taking that risk is most probably worth it due to the huge resources waste This change isn't effective yet - *Please reply here* if you agree or not to make this change available.
All of this is 'master' - 3.6 will follow if we will agree on activating that change.
CI is indeed many times a bottleneck mostly because of endless trivial rebases and I'm all in for this change, however, I would like to see how it works in master for a few weeks before we consider to implement the same change for the stable branches
Thanks, Roy
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
participants (4)
-
David Caro -
Eyal Edri -
Roy Golan -
Tal Nisan