Re: [ovirt-devel] local vdsm build fails

From: "David Caro" <dcaroest(a)redhat.com&gt; To: "Nir Soffer" <nsoffer(a)redhat.com&gt; Cc: "Piotr Kliczewski" <piotr.kliczewski(a)gmail.com&gt;, fsimonce(a)redhat.com, dcaro(a)redhat.com, devel(a)ovirt.org, "Dan Kenigsberg" <danken(a)redhat.com&gt; Sent: Friday, June 13, 2014 6:34:39 PM Subject: Re: [ovirt-devel] local vdsm build fails On Thu 12 Jun 2014 09:11:07 PM CEST, Nir Soffer wrote: > ----- Original Message ----- >> From: "David Caro" <dcaroest(a)redhat.com&gt; >> To: "Nir Soffer" <nsoffer(a)redhat.com&gt; >> Cc: "Piotr Kliczewski" <piotr.kliczewski(a)gmail.com&gt;, fsimonce(a)redhat.com, >> dcaro(a)redhat.com, devel(a)ovirt.org, "Dan >> Kenigsberg" <danken(a)redhat.com&gt; >> Sent: Thursday, June 12, 2014 9:48:54 PM >> Subject: Re: [ovirt-devel] local vdsm build fails >> >> On Thu 12 Jun 2014 12:47:11 PM CEST, Nir Soffer wrote: >>> ----- Original Message ----- >>>> From: "David Caro" <dcaroest(a)redhat.com&gt; >>>> To: "Nir Soffer" <nsoffer(a)redhat.com&gt; >>>> Cc: "Piotr Kliczewski" <piotr.kliczewski(a)gmail.com&gt;, >>>> fsimonce(a)redhat.com, >>>> dcaro(a)redhat.com, devel(a)ovirt.org, "Dan >>>> Kenigsberg" <danken(a)redhat.com&gt; >>>> Sent: Thursday, June 12, 2014 1:24:39 PM >>>> Subject: Re: [ovirt-devel] local vdsm build fails >>>> >>>> On Sun 08 Jun 2014 12:57:24 PM CEST, Nir Soffer wrote: >>>>> ----- Original Message ----- >>>>>> From: "David Caro" <dcaroest(a)redhat.com&gt; >>>>>> To: "Nir Soffer" <nsoffer(a)redhat.com&gt; >>>>>> Cc: "Piotr Kliczewski" <piotr.kliczewski(a)gmail.com&gt;, >>>>>> fsimonce(a)redhat.com, >>>>>> dcaro(a)redhat.com, devel(a)ovirt.org, "Dan >>>>>> Kenigsberg" <danken(a)redhat.com&gt; >>>>>> Sent: Friday, June 6, 2014 5:16:52 PM >>>>>> Subject: Re: [ovirt-devel] local vdsm build fails >>>>>> >>>>>> On Fri 06 Jun 2014 03:53:41 PM CEST, Nir Soffer wrote: >>>>>>> ----- Original Message ----- >>>>>>>> From: "Dan Kenigsberg" <danken(a)redhat.com&gt; >>>>>>>> To: "Piotr Kliczewski" <piotr.kliczewski(a)gmail.com&gt;, >>>>>>>> fsimonce(a)redhat.com, >>>>>>>> nsoffer(a)redhat.com, dcaro(a)redhat.com >>>>>>>> Cc: devel(a)ovirt.org >>>>>>>> Sent: Friday, June 6, 2014 12:15:18 PM >>>>>>>> Subject: Re: [ovirt-devel] local vdsm build fails >>>>>>>> >>>>>>>> On Fri, Jun 06, 2014 at 09:19:11AM +0200, Piotr Kliczewski wrote: >>>>>>>>> All, >>>>>>>>> >>>>>>>>> I pulled the latest vdsm from master and noticed that build is >>>>>>>>> failing. >>>>>>>>> >>>>>>>>> Here is the patch that causes the failuer: >>>>>>>>> >>>>>>>>> http://gerrit.ovirt.org/#/c/28226 >>>>>>>>> >>>>>>>>> and looking at jenkins comments I can see that jenkins was failing >>>>>>>>> with the same reason: >>>>>>>>> >>>>>>>>> http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs... >>>>>>> >>>>>>> Nir has already fix that as well. The storage tests were just fine, >>>>>>> but >>>>>>> a post build script was running cp incorrectly. >>>>>>> >>>>>>> David pointed that we need a way to distinguish between test errors >>>>>>> and >>>>>>> failures. >>>>>>> He suggested looking up strings in the test output - we should not go >>>>>>> there, unless >>>>>>> we want to "fix" this many more times in the future. >>>>>>> >>>>>>> I suggest to use the these rules: >>>>>>> >>>>>>> - SUCCESS - make check returns 0 >>>>>>> - FAILURE - make check returns 1 >>>>>>> - ERROR - anything else returned by make check or any other script. >>>>>>> >>>>>>> I think that make check does work like this, but it should be easy to >>>>>>> change. >>>>>>> >>>>>>> What do you think? >>>>>>> >>>>>>>> >>>>>>>> Thanks for your report. Nir has already fixed this in >>>>>>>> http://gerrit.ovirt.org/28426. >>>>>>>> >>>>>>>> It was introduced in http://gerrit.ovirt.org/#/c/28226/ but missed >>>>>>>> also >>>>>>>> because we have turned PYFLAKES off in unit test jobs. We must turn >>>>>>>> it >>>>>>>> on >>>>>>>> in >>>>>>>> at least one of the tests (or initiate a new jenkins job for `make >>>>>>>> check-local`). >>>>>>>> >>>>>>>> As a quick fix, David has re-enabled PYFLAKES in >>>>>>>> http://jenkins.ovirt.org/view/By%20Project/view/vdsm/job/vdsm_master_unit... >>>>>>>> >>>>>>>> Regards, >>>>>>>> Dan. >>>>>>>> >>>>>> >>>>>> Perfect for me, but you should know that it will fail also when >>>>>> strange >>>>>> things occur, for example, out of memory, of disk space, slave >>>>>> disconnected, network error, etc. >>>>>> >>>>>> If you are willing to treat those (the most common infra failures) as >>>>>> devel failures, then no problem on my side, >>>>> >>>>> I'm not - this is why we should separate test failures from test >>>>> errors. >>>>> >>>>>> but I don't want you to >>>>>> start ignoring test errors because it's most probably an infra error >>>>>> (don't get me wrong, it's totally normal to start ignoring an alarm >>>>>> that is not a real problem, as infra members we will try to minimize >>>>>> the infra issues, but it's not yet as stable as we'd like it to be). >>>>> >>>>> This is too late now, people are already ignoring jenkins reports >>>>> because >>>>> of the many false negatives :-) >>>> >>>> So the return code is not a good solution then, we have to see if it >>>> failed, and if it was due to an infra error or a devel error. I think >>>> that it's easier to filter for: >>>> >>>> * A string that means the tests did ran, probably at the end of the log >>>> so if there's a connection failure it will be detected as infra issue. >>>> * A string that identified if the test failed or passed >>>> >>>> And if none of those were found, then an infra failure is supposed. >>> >>> Ok, how about: >>> >>> 1. make check will write a file with test results - no other output >>> can go into that file so we don't have to use heuristics when >>> parsing the file. >>> >>> 2. If the file is found and parse successfully, tests either succeeded or >>> failed. >> What does 'parse successfully' mean? > > That I can open and understand the contents of this file. For example if I > expect the file to contain "PASS" or "FAILED" but it contains "BLAH" this > is not a test failure, this is a test error. So then you have passed from checking jenkins log output to check the contents of a file, I see no clear advantage. Nothing against though (it cleans up the jenkins log, and let's you compress the logs).

I'll need then the format of the file to check it.