10:46 a.m.
--SkvwRMAIpAhPCcCJ
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On 05/29 02:24, Nir Soffer wrote:
On Sun, May 29, 2016 at 2:10 AM, Nir Soffer
<nsoffer(a)redhat.com> wrote:
> It look like this when tests times out:
>
> 23:04:44 miscTests.EventTests
> 23:04:44 testEmit =
OK
> 23:04:44 testEmitCallbackException
=
OK
> 23:04:49 testEmitStale
=
OK
> 23:04:49 testInstanceMethod
=
OK
> 23:04:50 testInstanceMethodDead
=
OK
> 23:04:55 testOneShot
> 23:04:55 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
> 23:04:55 =3D Timeout completing tests - extracting
stacktrace
> =3D
> 23:04:55 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
> 23:04:55
> 23:04:55 attach: No such file or directory.
> 23:04:55 [New LWP 7887]
> 23:04:55 [New LWP 7880]
> 23:04:55 [New LWP 7873]
> 23:04:55 [New LWP 7866]
> 23:04:55 [New LWP 7859]
> 23:04:55 [New LWP 7852]
> 23:04:55 [New LWP 7845]
> 23:04:55 [Thread debugging using libthread_db enabled]
> 23:04:55 Using host libthread_db library "/lib64/libthread_db.so.1".
> 23:04:56 0x00007f17f0a1fa82 in pthread_cond_timedwait@(a)GLIBC_2.3.2 ()
> from /lib64/libpthread.so.0
> 23:04:56
> 23:04:56 Thread 8 (Thread 0x7f17df860700 (LWP 7845)):
> 23:04:56 Undefined command: "py-bt". Try "help".
> 23:04:56 OK
> 23:04:56 testUnregister
> 23:04:56 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
> 23:04:56 =3D Aborting tests
> =3D
> 23:04:56 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
> 23:04:56 ../tests/run_tests_local.sh: line 35: 7743 Killed
> "$PYTHON_EXE" ../tests/testrunner.py --local-modules $@
>
>
>
> On Sun, May 29, 2016 at 2:07 AM, Nir Soffer <nsoffer(a)redhat.com> wrote:
>> On Thu, May 26, 2016 at 11:08 PM, Nir Soffer <nsoffer(a)redhat.com> wrot=
e:
>>> Hi all,
>>>
>>> We had 2 issues causing vdsm check-patch and check-merge jobs to get =
stuck.
>>>
>>> I fixed the one that caused most trouble:
>>> https://gerrit.ovirt.org/57993
>>>
>>> The other issue may be related to ioprocess, I fixed a related issue:
>>> https://gerrit.ovirt.org/57473
>>>
>>> But I have seen stuck jobs after this change, so the issue may not
>>> be fixed yet.
>>>
>>> If you see a stuck vdsm job - job that run more than 15 minutes, plea=
se
>>> get me a backtrace:
>>>
>>> 1. locate the test_runner process pid:
>>>
>>> $ ps aux | grep testrunner.py | grep -v grep
>>> nsoffer 26297 82.6 0.9 389592 111144 pts/3 R+ 22:52 0:02
>>> /usr/bin/python ../tests/testrunner.py ...
>>>
>>> 2. save a backtrace:
>>>
>>> gdb attach 26297 --batch -ex "thread apply all py-bt" >
py-bt.out
>>
>> This requires the python-debuginfo package, typically installed using:
>>
>> dnf debuginfo-install python
>>
>> I sent this patch, detecting stuck vdsm tests, printing a backtrace, a=
nd
killing
>> the stuck process:
>> https://gerrit.ovirt.org/58212
>>
>> It works, but we don't get a backtrace, since python-debuginfo is not =
installed
>> although I require it - probably we need to add the
fedora-debug repos=
itory
>> to check-patch.repos. I tried to use the urls from
/etc/yum.repos.d/fe=
dora.repo,
>> but none of them work.
>>
>> I will need help from infra to get it working.
=20
I sent also this patch, that should fix the issue on jenkins, but I
cannot test it on jenkins:
https://gerrit.ovirt.org/58213
Instead of forcing adding the repo for all the projects, you should use the
*repos files that vdsm has in the automation directory to add there any ext=
ra
repos that you want when running/installing
=20
Nir
--=20
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605
Email: dcaro(a)redhat.com
IRC: dcaro|dcaroest@{freenode|oftc|redhat}
Web: www.redhat.com
RHT Global #: 82-62605
--SkvwRMAIpAhPCcCJ
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJXS+/nAAoJEEBxx+HSYmnDv1oH/jtM0tQ9G3j5Be3pUKRvB307
d8k9krvxfXKG8xaxZ+1qeMV0ix3Rlkj7o3Cm1GLbVdYd5mvsnbT86OGQMY0aRDKj
FLfFxOcbIsRgtM2DU6wQW8XujYlt6FD6WHDkIYSUlY7RddtXCLVteQP0rF8bwhvd
vu3QxI7/oK72GDoejfLkvB/XMU0NtXNCh/E9zCKJvZo9Wl+ORNJ6h8hmNYMPktUF
58dCxDhj25LGAFCokj/ILsTXQtTHiNEI++bTH4u557m089Hdz1HIQsZxvbrZOvd1
zlzJPuE2Yr+Szw0bxlAX+99gJNEklT+AZsDZQmnt/+GmWmAmGKrYdHRdI/WTLQc=
=PWhi
-----END PGP SIGNATURE-----
--SkvwRMAIpAhPCcCJ--