11:03 a.m.
----- Original Message -----
From: "David Caro" <dcaroest(a)redhat.com>
To: "Michal Skrivanek" <michal.skrivanek(a)redhat.com>
Cc: devel(a)ovirt.org
Sent: Friday, June 6, 2014 9:53:23 AM
Subject: Re: [ovirt-devel] local vdsm build fails
On Fri 06 Jun 2014 09:23:33 AM CEST, Michal Skrivanek wrote:
>
> On Jun 6, 2014, at 09:19 , Piotr Kliczewski <piotr.kliczewski(a)gmail.com>
> wrote:
>
>> All,
>>
>> I pulled the latest vdsm from master and noticed that build is failing.
>>
>> Here is the patch that causes the failuer:
>>
>> http://gerrit.ovirt.org/#/c/28226
>>
>> and looking at jenkins comments I can see that jenkins was failing
>> with the same reason:
>>
>>
http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs...
>
> btw at least yesterday again there were so many false errors with jenkins
> not being able to run the tests properly that it's unusable….
> was that the reason the result was ignored? (though the err is clear about
> relevance to that patch)
Can you point out which jobs were false positives? Also, can you
specify for each one how to determine if it's a test failure from the
logs? As specific as possible? We can filter the logs for those
failures and set a different message so you'll know from the gerrit
comments if it was a real issue or infra failure.
Quite some noise is from python segfaulting. I can reproduce the segfault locally
but I'm having hard time pinpointing the issue. Reported the bug upstream:
https://github.com/nose-devs/nose/issues/817
Let me summarize what I (we) know about python segfaulting:
* the segfault should be reproduceable on any box running nose >= 1.3.0, just using
$ cd vdsm
$ ./configure && make
$ NOSE_WITH_XUNIT=1 make check
or at least I can reproduce the issue on all the boxes I tried locally (vanilla F20, F19)
* if we run each testunit separately, we do NOT observe the failure.
This triggers the segfault:
$ cd tests
$ ./run_tests_local.sh ./*.py
This does not:
$ cd tests
$ for TEST in `ls ./*.py`; do ./run_tests_local.sh $TEST; done
* the stack traces I observed are huge, more than 750 levels deep.
This suggests the stack exausted, and this in turn probably triggered by some kind of
recursion
gone wild. Note the offending stack trace is just on one thread; all the others are
quiet.
* I tried to reproduce the issue with a simpler use case with no luck so far.
At the moment I don't have better suggestions than bite the bullet
and dig in the huge stack trace looking for repetitive patterns or some sort of hint.
A core dump available for post-mortem analysis, from my laptop, which is a F20 with few
updates
(mostly from virt-preview and few other places - full list, if relevant, provided as
pkgs.txt.gz in the
folder below)
here [link will be provided off-list]
core.20626.1000.gz is the fresh new core, core.20626.1000.md5 is its checksum.
--
Francesco Romani
RedHat Engineering Virtualization R & D
Phone: 8261328
IRC: fromani