--SkvwRMAIpAhPCcCJ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 05/29 02:24, Nir Soffer wrote:
On Sun, May 29, 2016 at 2:10 AM, Nir Soffer <nsoffer@redhat.com> wrote:
It look like this when tests times out:
23:04:44 miscTests.EventTests 23:04:44 testEmit = OK 23:04:44 testEmitCallbackException = OK 23:04:49 testEmitStale = OK 23:04:49 testInstanceMethod = OK 23:04:50 testInstanceMethodDead = OK 23:04:55 testOneShot 23:04:55 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D 23:04:55 =3D Timeout completing tests - extracting stacktrace =3D 23:04:55 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D 23:04:55 23:04:55 attach: No such file or directory. 23:04:55 [New LWP 7887] 23:04:55 [New LWP 7880] 23:04:55 [New LWP 7873] 23:04:55 [New LWP 7866] 23:04:55 [New LWP 7859] 23:04:55 [New LWP 7852] 23:04:55 [New LWP 7845] 23:04:55 [Thread debugging using libthread_db enabled] 23:04:55 Using host libthread_db library "/lib64/libthread_db.so.1". 23:04:56 0x00007f17f0a1fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 23:04:56 23:04:56 Thread 8 (Thread 0x7f17df860700 (LWP 7845)): 23:04:56 Undefined command: "py-bt". Try "help". 23:04:56 OK 23:04:56 testUnregister 23:04:56 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D 23:04:56 =3D Aborting tests =3D 23:04:56 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D 23:04:56 ../tests/run_tests_local.sh: line 35: 7743 Killed "$PYTHON_EXE" ../tests/testrunner.py --local-modules $@
On Sun, May 29, 2016 at 2:07 AM, Nir Soffer <nsoffer@redhat.com> wrote:
On Thu, May 26, 2016 at 11:08 PM, Nir Soffer <nsoffer@redhat.com> wrot= e:
Hi all,
We had 2 issues causing vdsm check-patch and check-merge jobs to get = stuck.
I fixed the one that caused most trouble: https://gerrit.ovirt.org/57993
The other issue may be related to ioprocess, I fixed a related issue: https://gerrit.ovirt.org/57473
But I have seen stuck jobs after this change, so the issue may not be fixed yet.
If you see a stuck vdsm job - job that run more than 15 minutes, plea= se get me a backtrace:
1. locate the test_runner process pid:
$ ps aux | grep testrunner.py | grep -v grep nsoffer 26297 82.6 0.9 389592 111144 pts/3 R+ 22:52 0:02 /usr/bin/python ../tests/testrunner.py ...
2. save a backtrace:
gdb attach 26297 --batch -ex "thread apply all py-bt" > py-bt.out
This requires the python-debuginfo package, typically installed using:
dnf debuginfo-install python
I sent this patch, detecting stuck vdsm tests, printing a backtrace, a= nd killing the stuck process: https://gerrit.ovirt.org/58212
It works, but we don't get a backtrace, since python-debuginfo is not = installed although I require it - probably we need to add the fedora-debug repos= itory to check-patch.repos. I tried to use the urls from /etc/yum.repos.d/fe= dora.repo, but none of them work.
I will need help from infra to get it working. =20 I sent also this patch, that should fix the issue on jenkins, but I cannot test it on jenkins: https://gerrit.ovirt.org/58213
Instead of forcing adding the repo for all the projects, you should use the *repos files that vdsm has in the automation directory to add there any ext= ra repos that you want when running/installing
=20 Nir
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605 --SkvwRMAIpAhPCcCJ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXS+/nAAoJEEBxx+HSYmnDv1oH/jtM0tQ9G3j5Be3pUKRvB307 d8k9krvxfXKG8xaxZ+1qeMV0ix3Rlkj7o3Cm1GLbVdYd5mvsnbT86OGQMY0aRDKj FLfFxOcbIsRgtM2DU6wQW8XujYlt6FD6WHDkIYSUlY7RddtXCLVteQP0rF8bwhvd vu3QxI7/oK72GDoejfLkvB/XMU0NtXNCh/E9zCKJvZo9Wl+ORNJ6h8hmNYMPktUF 58dCxDhj25LGAFCokj/ILsTXQtTHiNEI++bTH4u557m089Hdz1HIQsZxvbrZOvd1 zlzJPuE2Yr+Szw0bxlAX+99gJNEklT+AZsDZQmnt/+GmWmAmGKrYdHRdI/WTLQc= =PWhi -----END PGP SIGNATURE----- --SkvwRMAIpAhPCcCJ--