This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --JWv6SKN48HC4bWmJvJSBXV2crBPp428RU Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Thu 12 Jun 2014 09:11:07 PM CEST, Nir Soffer wrote:
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "Piotr Kliczewski" <piotr.kliczewski@gmail.com>, fsimonce@redhat.c= om, dcaro@redhat.com, devel@ovirt.org, "Dan Kenigsberg" <danken@redhat.com> Sent: Thursday, June 12, 2014 9:48:54 PM Subject: Re: [ovirt-devel] local vdsm build fails
On Thu 12 Jun 2014 12:47:11 PM CEST, Nir Soffer wrote:
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "Piotr Kliczewski" <piotr.kliczewski@gmail.com>, fsimonce@redhat= =2Ecom, dcaro@redhat.com, devel@ovirt.org, "Dan Kenigsberg" <danken@redhat.com> Sent: Thursday, June 12, 2014 1:24:39 PM Subject: Re: [ovirt-devel] local vdsm build fails
On Sun 08 Jun 2014 12:57:24 PM CEST, Nir Soffer wrote:
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "Piotr Kliczewski" <piotr.kliczewski@gmail.com>, fsimonce@redhat.com, dcaro@redhat.com, devel@ovirt.org, "Dan Kenigsberg" <danken@redhat.com> Sent: Friday, June 6, 2014 5:16:52 PM Subject: Re: [ovirt-devel] local vdsm build fails
On Fri 06 Jun 2014 03:53:41 PM CEST, Nir Soffer wrote: > ----- Original Message ----- >> From: "Dan Kenigsberg" <danken@redhat.com> >> To: "Piotr Kliczewski" <piotr.kliczewski@gmail.com>, >> fsimonce@redhat.com, >> nsoffer@redhat.com, dcaro@redhat.com >> Cc: devel@ovirt.org >> Sent: Friday, June 6, 2014 12:15:18 PM >> Subject: Re: [ovirt-devel] local vdsm build fails >> >> On Fri, Jun 06, 2014 at 09:19:11AM +0200, Piotr Kliczewski wrote= : >>> All, >>> >>> I pulled the latest vdsm from master and noticed that build is >>> failing. >>> >>> Here is the patch that causes the failuer: >>> >>> http://gerrit.ovirt.org/#/c/28226 >>> >>> and looking at jenkins comments I can see that jenkins was fail= ing >>> with the same reason: >>> >>> http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tes= ts_localfs_gerrit/1064/console > > Nir has already fix that as well. The storage tests were just fin= e, but > a post build script was running cp incorrectly. > > David pointed that we need a way to distinguish between test erro= rs and > failures. > He suggested looking up strings in the test output - we should no= t go > there, unless > we want to "fix" this many more times in the future. > > I suggest to use the these rules: > > - SUCCESS - make check returns 0 > - FAILURE - make check returns 1 > - ERROR - anything else returned by make check or any other scrip= t. > > I think that make check does work like this, but it should be eas= y to > change. > > What do you think? > >> >> Thanks for your report. Nir has already fixed this in >> http://gerrit.ovirt.org/28426. >> >> It was introduced in http://gerrit.ovirt.org/#/c/28226/ but miss= ed >> also >> because we have turned PYFLAKES off in unit test jobs. We must t= urn it >> on >> in >> at least one of the tests (or initiate a new jenkins job for `ma= ke >> check-local`). >> >> As a quick fix, David has re-enabled PYFLAKES in >> http://jenkins.ovirt.org/view/By%20Project/view/vdsm/job/vdsm_ma= ster_unit_tests/configure >> >> Regards, >> Dan. >>
Perfect for me, but you should know that it will fail also when st= range things occur, for example, out of memory, of disk space, slave disconnected, network error, etc.
If you are willing to treat those (the most common infra failures)= as devel failures, then no problem on my side,
I'm not - this is why we should separate test failures from test er= rors.
but I don't want you to start ignoring test errors because it's most probably an infra err= or (don't get me wrong, it's totally normal to start ignoring an alar= m that is not a real problem, as infra members we will try to minimi= ze the infra issues, but it's not yet as stable as we'd like it to be= ).
This is too late now, people are already ignoring jenkins reports b= ecause of the many false negatives :-)
So the return code is not a good solution then, we have to see if it=
failed, and if it was due to an infra error or a devel error. I thin= k that it's easier to filter for:
* A string that means the tests did ran, probably at the end of the = log so if there's a connection failure it will be detected as infra issu= e. * A string that identified if the test failed or passed
And if none of those were found, then an infra failure is supposed.
Ok, how about:
1. make check will write a file with test results - no other output can go into that file so we don't have to use heuristics when parsing the file.
2. If the file is found and parse successfully, tests either succeede= d or failed. What does 'parse successfully' mean?
That I can open and understand the contents of this file. For example i= f I expect the file to contain "PASS" or "FAILED" but it contains "BLAH" th= is is not a test failure, this is a test error.
So then you have passed from checking jenkins log output to check the=20 contents of a file, I see no clear advantage. Nothing against though=20 (it cleans up the jenkins log, and let's you compress the logs). I'll need then the format of the file to check it.
3. Any other failure is a test error - failure is *never* assumed
You mean an infra issue?
To make it clear:
- test success - all tests completed and passed
- test failure - all tests completed and some of them failed
- test error - anything else, meaning, I don't know if tests passed or =
failed.
This can be caused by bad tests, infra issue, anything. Practically t= his means that the test author and infra should understand the error and fix it= , otherwise the developer does not know if her code is good or bad.
Nir
-- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --JWv6SKN48HC4bWmJvJSBXV2crBPp428RU Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJTmxoPAAoJEEBxx+HSYmnDcN4H/RSD2ggcjH4sc6ui8aiqrP3/ x/paZZQqHa+ekyjKw2tIQR3qIyzMVzoSvhDnCz8hLzxdpFl0+ko+7YBHa31i+8DV 7Ar+0YpzxN5LaMf2mbgcfIBtahC1kTqm+GDHE0bVvWjMNu630PwUcB1D6i3IgbWb a0Mscaq8KSlFmEpr0z9AOScLaSWHimKAjok/6Gc06eOQmAGuecVPJaDSto4mBWw1 CKmZ95t/WF0CgH9yubvLhU7NNjjJl8h2FDNnxNnXLpqoCd53krAz6uxP6E9jY0Tm rAl2ptCFs2gnVKLGm2/fPi/IQuxYQnhVZ3j9R7m6iMx2ArCZtYi7DFr/Pp0Kkk4= =MmCo -----END PGP SIGNATURE----- --JWv6SKN48HC4bWmJvJSBXV2crBPp428RU--