1:48 p.m.
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--OUIhV6GJhKMmtM9DDKvaaOwtkNotjNpmD
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
On Thu 12 Jun 2014 12:47:11 PM CEST, Nir Soffer wrote:
----- Original Message -----
> From: "David Caro" <dcaroest(a)redhat.com>
> To: "Nir Soffer" <nsoffer(a)redhat.com>
> Cc: "Piotr Kliczewski" <piotr.kliczewski(a)gmail.com>,
fsimonce(a)redhat.c=
om, dcaro(a)redhat.com, devel(a)ovirt.org, "Dan
> Kenigsberg" <danken(a)redhat.com>
> Sent: Thursday, June 12, 2014 1:24:39 PM
> Subject: Re: [ovirt-devel] local vdsm build fails
>
> On Sun 08 Jun 2014 12:57:24 PM CEST, Nir Soffer wrote:
>> ----- Original Message -----
>>> From: "David Caro" <dcaroest(a)redhat.com>
>>> To: "Nir Soffer" <nsoffer(a)redhat.com>
>>> Cc: "Piotr Kliczewski" <piotr.kliczewski(a)gmail.com>,
fsimonce@redhat=
=2Ecom,
>>> dcaro(a)redhat.com, devel(a)ovirt.org, "Dan
>>> Kenigsberg" <danken(a)redhat.com>
>>> Sent: Friday, June 6, 2014 5:16:52 PM
>>> Subject: Re: [ovirt-devel] local vdsm build fails
>>>
>>> On Fri 06 Jun 2014 03:53:41 PM CEST, Nir Soffer wrote:
>>>> ----- Original Message -----
>>>>> From: "Dan Kenigsberg" <danken(a)redhat.com>
>>>>> To: "Piotr Kliczewski" <piotr.kliczewski(a)gmail.com>,
>>>>> fsimonce(a)redhat.com,
>>>>> nsoffer(a)redhat.com, dcaro(a)redhat.com
>>>>> Cc: devel(a)ovirt.org
>>>>> Sent: Friday, June 6, 2014 12:15:18 PM
>>>>> Subject: Re: [ovirt-devel] local vdsm build fails
>>>>>
>>>>> On Fri, Jun 06, 2014 at 09:19:11AM +0200, Piotr Kliczewski wrote:
>>>>>> All,
>>>>>>
>>>>>> I pulled the latest vdsm from master and noticed that build is
fa=
iling.
>>>>>>
>>>>>> Here is the patch that causes the failuer:
>>>>>>
>>>>>> http://gerrit.ovirt.org/#/c/28226
>>>>>>
>>>>>> and looking at jenkins comments I can see that jenkins was
failin=
g
>>>>>> with the same reason:
>>>>>>
>>>>>>
http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests=
_localfs_gerrit/1064/console
>>>>
>>>> Nir has already fix that as well. The storage tests were just fine,=
but
>>>> a post build script was running cp incorrectly.
>>>>
>>>> David pointed that we need a way to distinguish between test errors=
and
>>>> failures.
>>>> He suggested looking up strings in the test output - we should not =
go
>>>> there, unless
>>>> we want to "fix" this many more times in the future.
>>>>
>>>> I suggest to use the these rules:
>>>>
>>>> - SUCCESS - make check returns 0
>>>> - FAILURE - make check returns 1
>>>> - ERROR - anything else returned by make check or any other script.=
>>>>
>>>> I think that make check does work like this, but it should be easy =
to
>>>> change.
>>>>
>>>> What do you think?
>>>>
>>>>>
>>>>> Thanks for your report. Nir has already fixed this in
>>>>> http://gerrit.ovirt.org/28426.
>>>>>
>>>>> It was introduced in http://gerrit.ovirt.org/#/c/28226/ but missed=
also
>>>>> because we have turned PYFLAKES off in unit test
jobs. We must tur=
n it
>>>>> on
>>>>> in
>>>>> at least one of the tests (or initiate a new jenkins job for `make=
>>>>> check-local`).
>>>>>
>>>>> As a quick fix, David has re-enabled PYFLAKES in
>>>>> http://jenkins.ovirt.org/view/By%20Project/view/vdsm/job/vdsm_mast=
er_unit_tests/configure
>>>>>
>>>>> Regards,
>>>>> Dan.
>>>>>
>>>
>>> Perfect for me, but you should know that it will fail also when stra=
nge
>>> things occur, for example, out of memory, of disk space,
slave
>>> disconnected, network error, etc.
>>>
>>> If you are willing to treat those (the most common infra failures) a=
s
>>> devel failures, then no problem on my side,
>>
>> I'm not - this is why we should separate test failures from test erro=
rs.
>>
>>> but I don't want you to
>>> start ignoring test errors because it's most probably an infra error=
>>> (don't get me wrong, it's totally normal to start
ignoring an alarm
>>> that is not a real problem, as infra members we will try to minimize=
>>> the infra issues, but it's not yet as stable as
we'd like it to be).=
>>
>> This is too late now, people are already ignoring jenkins reports bec=
ause
>> of the many false negatives :-)
>
> So the return code is not a good solution then, we have to see if it
> failed, and if it was due to an infra error or a devel error. I think
> that it's easier to filter for:
>
> * A string that means the tests did ran, probably at the end of the lo=
g
> so if there's a connection failure it will be detected as
infra issue.=
> * A string that identified if the test failed or passed
>
> And if none of those were found, then an infra failure is supposed.
Ok, how about:
1. make check will write a file with test results - no other output
can go into that file so we don't have to use heuristics when
parsing the file.
2. If the file is found and parse successfully, tests either succeeded =
or
failed.
What does 'parse successfully' mean?
3. Any other failure is a test error - failure is *never* assumed
You mean an infra
issue?
--
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Email: dcaro(a)redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
--OUIhV6GJhKMmtM9DDKvaaOwtkNotjNpmD
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJTmfYWAAoJEEBxx+HSYmnD3csH/3thGlI3uE7ZYQPKkiVFQ9Dj
qCbPu7ITr+GBeENATwcGbmWc0F9wf9QilC//DcqjzYWRqC54ZOgNd9qpgIHIGu3o
4+EE9UWNVd4jdz2hvuoqDcyxjQ3FcX2uzyMMbO7Lap5WQCGE7D/wUCZIwPsDYWWP
sS91liMAUTPqmCYY7SFb/qaoFpgmnpZTboTJxRGHEOa1GCBHz80WgHTZSP0paHHK
g8SQXKJQLOtIsI2tCMoGwVvnP2pTHKnUVEEitho2Y71eGtvfY4FmXlNGjWeFrKpe
bzWWtX/G3DlVUpd20aues0sZp5Z9V0fKihAsW4HpsPcGWLVQeUA27XEJYvb9TnY=
=bR5A
-----END PGP SIGNATURE-----
--OUIhV6GJhKMmtM9DDKvaaOwtkNotjNpmD--