Hi all, We had 2 issues causing vdsm check-patch and check-merge jobs to get stuck. I fixed the one that caused most trouble: https://gerrit.ovirt.org/57993 The other issue may be related to ioprocess, I fixed a related issue: https://gerrit.ovirt.org/57473 But I have seen stuck jobs after this change, so the issue may not be fixed yet. If you see a stuck vdsm job - job that run more than 15 minutes, please get me a backtrace: 1. locate the test_runner process pid: $ ps aux | grep testrunner.py | grep -v grep nsoffer 26297 82.6 0.9 389592 111144 pts/3 R+ 22:52 0:02 /usr/bin/python ../tests/testrunner.py ... 2. save a backtrace: gdb attach 26297 --batch -ex "thread apply all py-bt" > py-bt.out Thanks, Nir
On Thu, May 26, 2016 at 11:08 PM, Nir Soffer <nsoffer@redhat.com> wrote:
Hi all,
We had 2 issues causing vdsm check-patch and check-merge jobs to get stuck.
I fixed the one that caused most trouble: https://gerrit.ovirt.org/57993
The other issue may be related to ioprocess, I fixed a related issue: https://gerrit.ovirt.org/57473
But I have seen stuck jobs after this change, so the issue may not be fixed yet.
If you see a stuck vdsm job - job that run more than 15 minutes, please get me a backtrace:
1. locate the test_runner process pid:
$ ps aux | grep testrunner.py | grep -v grep nsoffer 26297 82.6 0.9 389592 111144 pts/3 R+ 22:52 0:02 /usr/bin/python ../tests/testrunner.py ...
2. save a backtrace:
gdb attach 26297 --batch -ex "thread apply all py-bt" > py-bt.out
This requires the python-debuginfo package, typically installed using: dnf debuginfo-install python I sent this patch, detecting stuck vdsm tests, printing a backtrace, and killing the stuck process: https://gerrit.ovirt.org/58212 It works, but we don't get a backtrace, since python-debuginfo is not installed although I require it - probably we need to add the fedora-debug repository to check-patch.repos. I tried to use the urls from /etc/yum.repos.d/fedora.repo, but none of them work. I will need help from infra to get it working. Nir
It look like this when tests times out: 23:04:44 miscTests.EventTests 23:04:44 testEmit OK 23:04:44 testEmitCallbackException OK 23:04:49 testEmitStale OK 23:04:49 testInstanceMethod OK 23:04:50 testInstanceMethodDead OK 23:04:55 testOneShot 23:04:55 ======================================================================== 23:04:55 = Timeout completing tests - extracting stacktrace = 23:04:55 ======================================================================== 23:04:55 23:04:55 attach: No such file or directory. 23:04:55 [New LWP 7887] 23:04:55 [New LWP 7880] 23:04:55 [New LWP 7873] 23:04:55 [New LWP 7866] 23:04:55 [New LWP 7859] 23:04:55 [New LWP 7852] 23:04:55 [New LWP 7845] 23:04:55 [Thread debugging using libthread_db enabled] 23:04:55 Using host libthread_db library "/lib64/libthread_db.so.1". 23:04:56 0x00007f17f0a1fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 23:04:56 23:04:56 Thread 8 (Thread 0x7f17df860700 (LWP 7845)): 23:04:56 Undefined command: "py-bt". Try "help". 23:04:56 OK 23:04:56 testUnregister 23:04:56 ======================================================================== 23:04:56 = Aborting tests = 23:04:56 ======================================================================== 23:04:56 ../tests/run_tests_local.sh: line 35: 7743 Killed "$PYTHON_EXE" ../tests/testrunner.py --local-modules $@ On Sun, May 29, 2016 at 2:07 AM, Nir Soffer <nsoffer@redhat.com> wrote:
On Thu, May 26, 2016 at 11:08 PM, Nir Soffer <nsoffer@redhat.com> wrote:
Hi all,
We had 2 issues causing vdsm check-patch and check-merge jobs to get stuck.
I fixed the one that caused most trouble: https://gerrit.ovirt.org/57993
The other issue may be related to ioprocess, I fixed a related issue: https://gerrit.ovirt.org/57473
But I have seen stuck jobs after this change, so the issue may not be fixed yet.
If you see a stuck vdsm job - job that run more than 15 minutes, please get me a backtrace:
1. locate the test_runner process pid:
$ ps aux | grep testrunner.py | grep -v grep nsoffer 26297 82.6 0.9 389592 111144 pts/3 R+ 22:52 0:02 /usr/bin/python ../tests/testrunner.py ...
2. save a backtrace:
gdb attach 26297 --batch -ex "thread apply all py-bt" > py-bt.out
This requires the python-debuginfo package, typically installed using:
dnf debuginfo-install python
I sent this patch, detecting stuck vdsm tests, printing a backtrace, and killing the stuck process: https://gerrit.ovirt.org/58212
It works, but we don't get a backtrace, since python-debuginfo is not installed although I require it - probably we need to add the fedora-debug repository to check-patch.repos. I tried to use the urls from /etc/yum.repos.d/fedora.repo, but none of them work.
I will need help from infra to get it working.
Nir
On Sun, May 29, 2016 at 2:10 AM, Nir Soffer <nsoffer@redhat.com> wrote:
It look like this when tests times out:
23:04:44 miscTests.EventTests 23:04:44 testEmit OK 23:04:44 testEmitCallbackException OK 23:04:49 testEmitStale OK 23:04:49 testInstanceMethod OK 23:04:50 testInstanceMethodDead OK 23:04:55 testOneShot 23:04:55 ======================================================================== 23:04:55 = Timeout completing tests - extracting stacktrace = 23:04:55 ======================================================================== 23:04:55 23:04:55 attach: No such file or directory. 23:04:55 [New LWP 7887] 23:04:55 [New LWP 7880] 23:04:55 [New LWP 7873] 23:04:55 [New LWP 7866] 23:04:55 [New LWP 7859] 23:04:55 [New LWP 7852] 23:04:55 [New LWP 7845] 23:04:55 [Thread debugging using libthread_db enabled] 23:04:55 Using host libthread_db library "/lib64/libthread_db.so.1". 23:04:56 0x00007f17f0a1fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 23:04:56 23:04:56 Thread 8 (Thread 0x7f17df860700 (LWP 7845)): 23:04:56 Undefined command: "py-bt". Try "help". 23:04:56 OK 23:04:56 testUnregister 23:04:56 ======================================================================== 23:04:56 = Aborting tests = 23:04:56 ======================================================================== 23:04:56 ../tests/run_tests_local.sh: line 35: 7743 Killed "$PYTHON_EXE" ../tests/testrunner.py --local-modules $@
On Sun, May 29, 2016 at 2:07 AM, Nir Soffer <nsoffer@redhat.com> wrote:
On Thu, May 26, 2016 at 11:08 PM, Nir Soffer <nsoffer@redhat.com> wrote:
Hi all,
We had 2 issues causing vdsm check-patch and check-merge jobs to get stuck.
I fixed the one that caused most trouble: https://gerrit.ovirt.org/57993
The other issue may be related to ioprocess, I fixed a related issue: https://gerrit.ovirt.org/57473
But I have seen stuck jobs after this change, so the issue may not be fixed yet.
If you see a stuck vdsm job - job that run more than 15 minutes, please get me a backtrace:
1. locate the test_runner process pid:
$ ps aux | grep testrunner.py | grep -v grep nsoffer 26297 82.6 0.9 389592 111144 pts/3 R+ 22:52 0:02 /usr/bin/python ../tests/testrunner.py ...
2. save a backtrace:
gdb attach 26297 --batch -ex "thread apply all py-bt" > py-bt.out
This requires the python-debuginfo package, typically installed using:
dnf debuginfo-install python
I sent this patch, detecting stuck vdsm tests, printing a backtrace, and killing the stuck process: https://gerrit.ovirt.org/58212
It works, but we don't get a backtrace, since python-debuginfo is not installed although I require it - probably we need to add the fedora-debug repository to check-patch.repos. I tried to use the urls from /etc/yum.repos.d/fedora.repo, but none of them work.
I will need help from infra to get it working.
I sent also this patch, that should fix the issue on jenkins, but I cannot test it on jenkins: https://gerrit.ovirt.org/58213 Nir
--SkvwRMAIpAhPCcCJ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 05/29 02:24, Nir Soffer wrote:
On Sun, May 29, 2016 at 2:10 AM, Nir Soffer <nsoffer@redhat.com> wrote:
It look like this when tests times out:
23:04:44 miscTests.EventTests 23:04:44 testEmit = OK 23:04:44 testEmitCallbackException = OK 23:04:49 testEmitStale = OK 23:04:49 testInstanceMethod = OK 23:04:50 testInstanceMethodDead = OK 23:04:55 testOneShot 23:04:55 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D 23:04:55 =3D Timeout completing tests - extracting stacktrace =3D 23:04:55 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D 23:04:55 23:04:55 attach: No such file or directory. 23:04:55 [New LWP 7887] 23:04:55 [New LWP 7880] 23:04:55 [New LWP 7873] 23:04:55 [New LWP 7866] 23:04:55 [New LWP 7859] 23:04:55 [New LWP 7852] 23:04:55 [New LWP 7845] 23:04:55 [Thread debugging using libthread_db enabled] 23:04:55 Using host libthread_db library "/lib64/libthread_db.so.1". 23:04:56 0x00007f17f0a1fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 23:04:56 23:04:56 Thread 8 (Thread 0x7f17df860700 (LWP 7845)): 23:04:56 Undefined command: "py-bt". Try "help". 23:04:56 OK 23:04:56 testUnregister 23:04:56 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D 23:04:56 =3D Aborting tests =3D 23:04:56 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D 23:04:56 ../tests/run_tests_local.sh: line 35: 7743 Killed "$PYTHON_EXE" ../tests/testrunner.py --local-modules $@
On Sun, May 29, 2016 at 2:07 AM, Nir Soffer <nsoffer@redhat.com> wrote:
On Thu, May 26, 2016 at 11:08 PM, Nir Soffer <nsoffer@redhat.com> wrot= e:
Hi all,
We had 2 issues causing vdsm check-patch and check-merge jobs to get = stuck.
I fixed the one that caused most trouble: https://gerrit.ovirt.org/57993
The other issue may be related to ioprocess, I fixed a related issue: https://gerrit.ovirt.org/57473
But I have seen stuck jobs after this change, so the issue may not be fixed yet.
If you see a stuck vdsm job - job that run more than 15 minutes, plea= se get me a backtrace:
1. locate the test_runner process pid:
$ ps aux | grep testrunner.py | grep -v grep nsoffer 26297 82.6 0.9 389592 111144 pts/3 R+ 22:52 0:02 /usr/bin/python ../tests/testrunner.py ...
2. save a backtrace:
gdb attach 26297 --batch -ex "thread apply all py-bt" > py-bt.out
This requires the python-debuginfo package, typically installed using:
dnf debuginfo-install python
I sent this patch, detecting stuck vdsm tests, printing a backtrace, a= nd killing the stuck process: https://gerrit.ovirt.org/58212
It works, but we don't get a backtrace, since python-debuginfo is not = installed although I require it - probably we need to add the fedora-debug repos= itory to check-patch.repos. I tried to use the urls from /etc/yum.repos.d/fe= dora.repo, but none of them work.
I will need help from infra to get it working. =20 I sent also this patch, that should fix the issue on jenkins, but I cannot test it on jenkins: https://gerrit.ovirt.org/58213
Instead of forcing adding the repo for all the projects, you should use the *repos files that vdsm has in the automation directory to add there any ext= ra repos that you want when running/installing
=20 Nir
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605 --SkvwRMAIpAhPCcCJ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXS+/nAAoJEEBxx+HSYmnDv1oH/jtM0tQ9G3j5Be3pUKRvB307 d8k9krvxfXKG8xaxZ+1qeMV0ix3Rlkj7o3Cm1GLbVdYd5mvsnbT86OGQMY0aRDKj FLfFxOcbIsRgtM2DU6wQW8XujYlt6FD6WHDkIYSUlY7RddtXCLVteQP0rF8bwhvd vu3QxI7/oK72GDoejfLkvB/XMU0NtXNCh/E9zCKJvZo9Wl+ORNJ6h8hmNYMPktUF 58dCxDhj25LGAFCokj/ILsTXQtTHiNEI++bTH4u557m089Hdz1HIQsZxvbrZOvd1 zlzJPuE2Yr+Szw0bxlAX+99gJNEklT+AZsDZQmnt/+GmWmAmGKrYdHRdI/WTLQc= =PWhi -----END PGP SIGNATURE----- --SkvwRMAIpAhPCcCJ--
On Mon, May 30, 2016 at 10:46 AM, David Caro Estevez <dcaro@redhat.com> wrote:
On 05/29 02:24, Nir Soffer wrote:
On Sun, May 29, 2016 at 2:10 AM, Nir Soffer <nsoffer@redhat.com> wrote:
It look like this when tests times out:
23:04:44 miscTests.EventTests 23:04:44 testEmit OK 23:04:44 testEmitCallbackException OK 23:04:49 testEmitStale OK 23:04:49 testInstanceMethod OK 23:04:50 testInstanceMethodDead OK 23:04:55 testOneShot 23:04:55 ======================================================================== 23:04:55 = Timeout completing tests - extracting stacktrace = 23:04:55 ======================================================================== 23:04:55 23:04:55 attach: No such file or directory. 23:04:55 [New LWP 7887] 23:04:55 [New LWP 7880] 23:04:55 [New LWP 7873] 23:04:55 [New LWP 7866] 23:04:55 [New LWP 7859] 23:04:55 [New LWP 7852] 23:04:55 [New LWP 7845] 23:04:55 [Thread debugging using libthread_db enabled] 23:04:55 Using host libthread_db library "/lib64/libthread_db.so.1". 23:04:56 0x00007f17f0a1fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 23:04:56 23:04:56 Thread 8 (Thread 0x7f17df860700 (LWP 7845)): 23:04:56 Undefined command: "py-bt". Try "help". 23:04:56 OK 23:04:56 testUnregister 23:04:56 ======================================================================== 23:04:56 = Aborting tests = 23:04:56 ======================================================================== 23:04:56 ../tests/run_tests_local.sh: line 35: 7743 Killed "$PYTHON_EXE" ../tests/testrunner.py --local-modules $@
On Sun, May 29, 2016 at 2:07 AM, Nir Soffer <nsoffer@redhat.com> wrote:
On Thu, May 26, 2016 at 11:08 PM, Nir Soffer <nsoffer@redhat.com> wrote:
Hi all,
We had 2 issues causing vdsm check-patch and check-merge jobs to get stuck.
I fixed the one that caused most trouble: https://gerrit.ovirt.org/57993
The other issue may be related to ioprocess, I fixed a related issue: https://gerrit.ovirt.org/57473
But I have seen stuck jobs after this change, so the issue may not be fixed yet.
If you see a stuck vdsm job - job that run more than 15 minutes, please get me a backtrace:
1. locate the test_runner process pid:
$ ps aux | grep testrunner.py | grep -v grep nsoffer 26297 82.6 0.9 389592 111144 pts/3 R+ 22:52 0:02 /usr/bin/python ../tests/testrunner.py ...
2. save a backtrace:
gdb attach 26297 --batch -ex "thread apply all py-bt" > py-bt.out
This requires the python-debuginfo package, typically installed using:
dnf debuginfo-install python
I sent this patch, detecting stuck vdsm tests, printing a backtrace, and killing the stuck process: https://gerrit.ovirt.org/58212
It works, but we don't get a backtrace, since python-debuginfo is not installed although I require it - probably we need to add the fedora-debug repository to check-patch.repos. I tried to use the urls from /etc/yum.repos.d/fedora.repo, but none of them work.
I will need help from infra to get it working.
I sent also this patch, that should fix the issue on jenkins, but I cannot test it on jenkins: https://gerrit.ovirt.org/58213
Instead of forcing adding the repo for all the projects, you should use the *repos files that vdsm has in the automation directory to add there any extra repos that you want when running/installing
Ok, I'm using now debuginfo-install to install the package - automatically enable the debuginfo repo, and it works on fedora. On el7, we fail to install the debuginfo package - I don't know why: 09:03:43 + debuginfo-install -y python 09:03:45 Could not find debuginfo for main pkg: python-2.7.5-34.el7.x86_64 09:03:45 Could not find debuginfo pkg for dependency package glibc-2.17-106.el7_2.6.x86_64 09:03:45 Could not find debuginfo pkg for dependency package python-libs-2.7.5-34.el7.x86_64 Maybe a missing repo? On fc23, we get the debuginfo package and we get a backtrace: 09:06:05 testlibTests.TestStuckProcess 09:09:19 test_stuck 09:09:19 ======================================================================== 09:09:19 = Timeout completing tests - extracting stacktrace = 09:09:19 ======================================================================== 09:09:19 09:09:19 attach: No such file or directory. 09:09:19 [New LWP 50476] 09:09:19 [New LWP 50469] 09:09:19 [New LWP 50462] 09:09:19 [New LWP 50455] 09:09:19 [New LWP 50448] 09:09:19 [New LWP 50441] 09:09:19 [New LWP 50434] 09:09:19 [New LWP 50427] 09:09:19 [New LWP 50420] 09:09:19 [New LWP 50413] 09:09:19 [New LWP 50406] 09:09:19 [New LWP 50399] 09:09:19 [New LWP 50392] 09:09:19 [New LWP 50385] 09:09:19 [New LWP 50378] 09:09:19 [New LWP 50371] 09:09:19 [New LWP 50364] 09:09:19 [New LWP 50247] 09:09:19 [New LWP 50240] 09:09:19 [New LWP 50233] 09:09:19 [New LWP 50226] 09:09:19 [New LWP 50219] 09:09:19 [New LWP 50216] 09:09:19 [New LWP 50189] 09:09:19 [New LWP 50175] 09:09:19 [New LWP 50161] 09:09:19 [New LWP 50147] 09:09:19 [New LWP 50133] 09:09:19 [New LWP 50119] 09:09:19 [New LWP 50105] 09:09:19 [New LWP 50091] 09:09:19 [New LWP 49889] 09:09:19 [New LWP 49858] 09:09:19 [Thread debugging using libthread_db enabled] 09:09:19 Using host libthread_db library "/lib64/libthread_db.so.1". 09:09:20 0x00007fb5ffece8a3 in select () at ../sysdeps/unix/syscall-template.S:84 09:09:20 84 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) 09:09:20 09:09:20 Thread 34 (Thread 0x7fb5ab7fe700 (LWP 49858)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 33 (Thread 0x7fb5a9899700 (LWP 49889)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 32 (Thread 0x7fb58effd700 (LWP 50091)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 31 (Thread 0x7fb58e7fc700 (LWP 50105)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 30 (Thread 0x7fb58dffb700 (LWP 50119)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 29 (Thread 0x7fb58d7fa700 (LWP 50133)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 28 (Thread 0x7fb58cff9700 (LWP 50147)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 27 (Thread 0x7fb56bfff700 (LWP 50161)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 26 (Thread 0x7fb56b7fe700 (LWP 50175)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 25 (Thread 0x7fb56affd700 (LWP 50189)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 24 (Thread 0x7fb59affd700 (LWP 50216)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 23 (Thread 0x7fb5e905f700 (LWP 50219)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 22 (Thread 0x7fb59bfff700 (LWP 50226)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 21 (Thread 0x7fb5e885e700 (LWP 50233)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 20 (Thread 0x7fb5e9860700 (LWP 50240)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 19 (Thread 0x7fb5eb864700 (LWP 50247)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 18 (Thread 0x7fb5c9061700 (LWP 50364)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 17 (Thread 0x7fb5cb7fe700 (LWP 50371)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 16 (Thread 0x7fb5cbfff700 (LWP 50378)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 15 (Thread 0x7fb5ea862700 (LWP 50385)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 14 (Thread 0x7fb5eb063700 (LWP 50392)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 13 (Thread 0x7fb5aaffd700 (LWP 50399)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 12 (Thread 0x7fb5a8f18700 (LWP 50406)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 11 (Thread 0x7fb59b7fe700 (LWP 50413)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 10 (Thread 0x7fb59a7fc700 (LWP 50420)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 9 (Thread 0x7fb599ffb700 (LWP 50427)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 8 (Thread 0x7fb5997fa700 (LWP 50434)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 7 (Thread 0x7fb598ff9700 (LWP 50441)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 6 (Thread 0x7fb58ffff700 (LWP 50448)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 5 (Thread 0x7fb58f7fe700 (LWP 50455)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 4 (Thread 0x7fb56a7fc700 (LWP 50462)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 3 (Thread 0x7fb569ffb700 (LWP 50469)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 2 (Thread 0x7fb5697fa700 (LWP 50476)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 1 (Thread 0x7fb60109d700 (LWP 49417)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testlibTests.py", line 264, in test_stuck 09:09:20 time.sleep(300) 09:09:20 File "/usr/lib64/python2.7/unittest/case.py", line 367, in run 09:09:20 testMethod() 09:09:20 File "/usr/lib64/python2.7/unittest/case.py", line 431, in __call__ 09:09:20 return self.run(*args, **kwds) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest 09:09:20 test(result) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run 09:09:20 self.runTest(result) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/case.py", line 45, in __call__ 09:09:20 return self.run(*arg, **kwarg) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 224, in run 09:09:20 test(orig) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 177, in __call__ 09:09:20 return self.run(*arg, **kw) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 224, in run 09:09:20 test(orig) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 177, in __call__ 09:09:20 return self.run(*arg, **kw) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 224, in run 09:09:20 test(orig) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 177, in __call__ 09:09:20 return self.run(*arg, **kw) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 224, in run 09:09:20 test(orig) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 177, in __call__ 09:09:20 return self.run(*arg, **kw) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line 62, in run 09:09:20 test(result) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line 207, in runTests 09:09:20 result = self.testRunner.run(self.test) 09:09:20 File "/usr/lib64/python2.7/unittest/main.py", line 95, in __init__ 09:09:20 self.runTests() 09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line 121, in __init__ 09:09:20 **extra_args) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line 301, in run 09:09:20 return TestProgram(*arg, **kw).success 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testlib.py", line 396, in run 09:09:20 sys.exit(not core.run(config=conf, testRunner=runner, argv=argv)) 09:09:20 File "../tests/testrunner.py", line 88, in <module> 09:09:20 testlib.run() 09:09:20 09:09:20 ======================================================================== 09:09:20 = Aborting tests = 09:09:20 ========================================================================
Nir
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
participants (2)
-
David Caro Estevez -
Nir Soffer