Jenkins is acting again - scoring patch when test was not run

[Nir Soffer]

----- Original Message -----
> From: "Eyal Edri" <eedri at redhat.com>
> To: "Nir Soffer" <nsoffer at redhat.com>
> Cc: "infra" <infra at ovirt.org>
> Sent: Sunday, March 2, 2014 10:26:51 PM
> Subject: Re: Jenkins is acting again - scoring patch when test was not run
> 
> 
> 
> ----- Original Message -----
> > From: "Nir Soffer" <nsoffer at redhat.com>
> > To: "infra" <infra at ovirt.org>
> > Sent: Friday, February 28, 2014 9:59:39 PM
> > Subject: Jenkins is acting again - scoring patch when test was not run
> > 
> > Hi infra,
> > 
> > Jenkins gave my patch -1 because it Jenkins was broken:
> > http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/7364/console
> > 
> > This is a major error; Jenkins should -1 a patch only if the it could run
> > the
> > relevant test, and the test failed.
> 
> I've changed the default behaviour of gerrit trigger to only give -1 on
> 'unstable'
> builds, and not on 'failed' builds.

What is unstable build?

> 
> builds in general should not fail (red) on real errors (if they do, we should
> fix
> relevant job), they should turn unstable (yellow).
> 
> and job won't turn unstable on an unexpected infra error (like happened now).
> 
> 
> > 
> > In this case the relevant test is running the pep8 tool, and scoring the
> > patch
> > is correct if the tool returned with non-zero exit code.
> > 
> > Any other failure is CI error and should not effect the patch score.
> > 
> > If you can, it would be nice if jenkins would add a comment on gerrit,
> > telling
> > about errors. In this case, the status should be ERROR instead of FAILURE.
> 
> i think it will happen now, please let us know if this happens again,
> we have the option to modify the default msg per failure.

You can test this by adding a fake error in some build script, and check
that it did not -1 the related patch.

Thanks
Nir