Jenkins is acting again - scoring patch when test was not run
Hi infra, Jenkins gave my patch -1 because it Jenkins was broken: http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/7364/console This is a major error; Jenkins should -1 a patch only if the it could run the relevant test, and the test failed. In this case the relevant test is running the pep8 tool, and scoring the patch is correct if the tool returned with non-zero exit code. Any other failure is CI error and should not effect the patch score. If you can, it would be nice if jenkins would add a comment on gerrit, telling about errors. In this case, the status should be ERROR instead of FAILURE. Examples: When jenkins fails to run a test: - http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/xxxx/ : ERROR When tests was run and failed: - http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/xxxx/ : FAILURE When test was run and succeeded: - http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/xxxx/ : SUCCESS Thanks, Nir
----- Original Message -----
From: "Nir Soffer" <nsoffer@redhat.com> To: "infra" <infra@ovirt.org> Sent: Friday, February 28, 2014 9:59:39 PM Subject: Jenkins is acting again - scoring patch when test was not run
Hi infra,
Jenkins gave my patch -1 because it Jenkins was broken: http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/7364/console
This is a major error; Jenkins should -1 a patch only if the it could run the relevant test, and the test failed.
I've changed the default behaviour of gerrit trigger to only give -1 on 'unstable' builds, and not on 'failed' builds. builds in general should not fail (red) on real errors (if they do, we should fix relevant job), they should turn unstable (yellow). and job won't turn unstable on an unexpected infra error (like happened now).
In this case the relevant test is running the pep8 tool, and scoring the patch is correct if the tool returned with non-zero exit code.
Any other failure is CI error and should not effect the patch score.
If you can, it would be nice if jenkins would add a comment on gerrit, telling about errors. In this case, the status should be ERROR instead of FAILURE.
i think it will happen now, please let us know if this happens again, we have the option to modify the default msg per failure.
Examples:
When jenkins fails to run a test: - http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/xxxx/ : ERROR
When tests was run and failed: - http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/xxxx/ : FAILURE
When test was run and succeeded: - http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/xxxx/ : SUCCESS
Thanks, Nir _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "infra" <infra@ovirt.org> Sent: Sunday, March 2, 2014 10:26:51 PM Subject: Re: Jenkins is acting again - scoring patch when test was not run
----- Original Message -----
From: "Nir Soffer" <nsoffer@redhat.com> To: "infra" <infra@ovirt.org> Sent: Friday, February 28, 2014 9:59:39 PM Subject: Jenkins is acting again - scoring patch when test was not run
Hi infra,
Jenkins gave my patch -1 because it Jenkins was broken: http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/7364/console
This is a major error; Jenkins should -1 a patch only if the it could run the relevant test, and the test failed.
I've changed the default behaviour of gerrit trigger to only give -1 on 'unstable' builds, and not on 'failed' builds.
What is unstable build?
builds in general should not fail (red) on real errors (if they do, we should fix relevant job), they should turn unstable (yellow).
and job won't turn unstable on an unexpected infra error (like happened now).
In this case the relevant test is running the pep8 tool, and scoring the patch is correct if the tool returned with non-zero exit code.
Any other failure is CI error and should not effect the patch score.
If you can, it would be nice if jenkins would add a comment on gerrit, telling about errors. In this case, the status should be ERROR instead of FAILURE.
i think it will happen now, please let us know if this happens again, we have the option to modify the default msg per failure.
You can test this by adding a fake error in some build script, and check that it did not -1 the related patch. Thanks Nir
----- Original Message -----
From: "Nir Soffer" <nsoffer@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "infra" <infra@ovirt.org> Sent: Sunday, March 2, 2014 10:37:42 PM Subject: Re: Jenkins is acting again - scoring patch when test was not run
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "infra" <infra@ovirt.org> Sent: Sunday, March 2, 2014 10:26:51 PM Subject: Re: Jenkins is acting again - scoring patch when test was not run
----- Original Message -----
From: "Nir Soffer" <nsoffer@redhat.com> To: "infra" <infra@ovirt.org> Sent: Friday, February 28, 2014 9:59:39 PM Subject: Jenkins is acting again - scoring patch when test was not run
Hi infra,
Jenkins gave my patch -1 because it Jenkins was broken: http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/7364/console
This is a major error; Jenkins should -1 a patch only if the it could run the relevant test, and the test failed.
I've changed the default behaviour of gerrit trigger to only give -1 on 'unstable' builds, and not on 'failed' builds.
What is unstable build?
this is a build status that indicates the job logic has failed (e.g unit test failed/pep8 check/etc...), a build won't be on unstable status unless it is set to do so on specific logic, leaving all unexpected infra issues to finish as 'failure' (red).
builds in general should not fail (red) on real errors (if they do, we should fix relevant job), they should turn unstable (yellow).
and job won't turn unstable on an unexpected infra error (like happened now).
In this case the relevant test is running the pep8 tool, and scoring the patch is correct if the tool returned with non-zero exit code.
Any other failure is CI error and should not effect the patch score.
If you can, it would be nice if jenkins would add a comment on gerrit, telling about errors. In this case, the status should be ERROR instead of FAILURE.
i think it will happen now, please let us know if this happens again, we have the option to modify the default msg per failure.
You can test this by adding a fake error in some build script, and check that it did not -1 the related patch.
works - http://gerrit.ovirt.org/#/c/25254/ failed on infra issue, but patch didn't get -1.
Thanks Nir
On Fri, Feb 28, 2014 at 02:59:39PM -0500, Nir Soffer wrote:
Hi infra,
Jenkins gave my patch -1 because it Jenkins was broken: http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/7364/console
This is a major error; Jenkins should -1 a patch only if the it could run the relevant test, and the test failed.
In this case the relevant test is running the pep8 tool, and scoring the patch is correct if the tool returned with non-zero exit code.
Any other failure is CI error and should not effect the patch score.
If you can, it would be nice if jenkins would add a comment on gerrit, telling about errors. In this case, the status should be ERROR instead of FAILURE.
Examples:
When jenkins fails to run a test: - http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/xxxx/ : ERROR
When tests was run and failed: - http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/xxxx/ : FAILURE
When test was run and succeeded: - http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/xxxx/ : SUCCESS
Hear hear!
participants (3)
-
Dan Kenigsberg -
Eyal Edri -
Nir Soffer