Test succeeded, hudson failed, test marked as FAILURE in gerrit

[Nir Soffer]

----- Original Message -----
> From: "David Caro" <dcaroest at redhat.com>
> To: "Nir Soffer" <nsoffer at redhat.com>
> Cc: "infra" <infra at ovirt.org>
> Sent: Tuesday, July 1, 2014 9:03:12 PM
> Subject: Re: Test succeeded, hudson failed, test marked as FAILURE in gerrit
> 
> On Tue 01 Jul 2014 06:33:08 PM CEST, Nir Soffer wrote:
> > Hi infra,
> >
> > Looking in the console log
> > http://jenkins.ovirt.org/job/vdsm_master_unit-tests_created/10774/console
> >
> > We can see that the test succeeded:
> >
> >     Ran 551 tests in 264.511s
> >
> >     OK (SKIP=41)
> >
> > But hudson failed:
> >
> >     FATAL: hudson.remoting.RequestAbortedException:
> >     hudson.remoting.Channel$OrderlyShutdown:
> >     java.util.concurrent.TimeoutException: Ping started on 1404230202100
> >     hasn't completed at 1404230442456
> >
> > And the test was marked as FAILURE:
> >
> >     http://jenkins.ovirt.org/job/vdsm_master_unit-tests_created/10774/ :
> >     FAILURE
> >
> > This is a recurring problem in the CI. We need to eliminate these errors
> > which
> > waste developers time, and worse, cause them to ignore failures.
> >
> > Please give this issue higher priority.
> >
> > Thanks,
> > Nir
> > _______________________________________________
> > Infra mailing list
> > Infra at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/infra
> 
> Not long ago we started a thread to try and determine if a job failed
> due to test error (meaning, one of the tests failed) or infra error
> (anything else, for example, network error, timeout...).
> iirc the last proposal was to generate a test file with a strict format
> that the job will check, and if not found or malformed it will be
> assumed infra issue, and if well formed, the file itself would
> determine if it passed of there were any test errors.
> 
> But no specific format was proposed, or file for that matter. I would
> be glad with just two messages (very strict format) in the log, tests
> ran and passed, and tests ran and did not pass. We are already doing
> that on a few jobs with good results for other projects.
> So if you want this to get pushed, please, feel free to provide a
> deterministic way for ci to detect if the tests did run properly and
> the result of the run. This also wastes infra time so please provide a
> solution.
> 
> Also, can you open a ticket on https://fedorahosted.org/ovirt/newticket
> for any specific request? If it's not there, it has no priority. We are
> trying to open there all the issues so we can manage them better and
> leave the list for discussions and announcements.

Right - we need continue with this.

I suggest we make a call tomorrow to discuss the solution.

Nir