<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 1, 2016 at 1:44 PM, Nir Soffer <span dir="ltr"><<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi all,<br>
<br>
So we have this patch, aborting stuck tests and printing a backtrace<br>
of all threads:<br>
<a href="https://gerrit.ovirt.org/#/c/58212" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/58212</a><br>
<br>
On fedora, we get now a nice backtrace (see bellow) - which show that<br>
we have unexpected<br>
threads running during the tests.<br>
<br>
we have 25 iopproces threads like this:<br>
<br>
09:09:20 Thread 34 (Thread 0x7fb5ab7fe700 (LWP 49858)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
<br>
This is expected given the current code managing these, and are daemon threads.<br>
<br>
We have 8 reactor threads:<br>
<br>
09:09:20 Thread 32 (Thread 0x7fb58effd700 (LWP 50091)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2<br>
09:09:20 r = pollster.poll(timeout)<br>
09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop<br>
09:09:20 poll_fun(timeout, map)<br>
09:09:20 File<br>
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py",<br>
line 212, in process_requests<br>
09:09:20 count=1,<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
<br>
I don't know where these are started and it they are daemon threads.<br>
<br>
Searching the tests, we have too much code creating threads directly,<br>
instead of using vdsm.concurrent.thread:<br>
<br>
$ git grep Thread tests/ | wc -l<br>
79<br>
<br>
Please check your tests, and replace all threads to use testlib.start_thread()<br>
creating daemon threads.<br>
<br>
Code inheriting from threading.Thread is the worst, please keep a thread<br>
instance instead, and start it with testlib.start_thread().<br>
<br>
I sent this patch fixing sslTests for now:<br>
<a href="https://gerrit.ovirt.org/58435" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/58435</a><br>
<br>
Nir<br>
<br>
----<br>
<br>
09:06:05 testlibTests.TestStuckProcess<br>
09:09:19 test_stuck<br>
09:09:19 ========================================================================<br>
09:09:19 = Timeout completing tests - extracting stacktrace<br>
=<br>
09:09:19 ========================================================================<br>
09:09:19<br>
09:09:19 attach: No such file or directory.<br>
09:09:19 [New LWP 50476]<br>
09:09:19 [New LWP 50469]<br>
09:09:19 [New LWP 50462]<br>
09:09:19 [New LWP 50455]<br>
09:09:19 [New LWP 50448]<br>
09:09:19 [New LWP 50441]<br>
09:09:19 [New LWP 50434]<br>
09:09:19 [New LWP 50427]<br>
09:09:19 [New LWP 50420]<br>
09:09:19 [New LWP 50413]<br>
09:09:19 [New LWP 50406]<br>
09:09:19 [New LWP 50399]<br>
09:09:19 [New LWP 50392]<br>
09:09:19 [New LWP 50385]<br>
09:09:19 [New LWP 50378]<br>
09:09:19 [New LWP 50371]<br>
09:09:19 [New LWP 50364]<br>
09:09:19 [New LWP 50247]<br>
09:09:19 [New LWP 50240]<br>
09:09:19 [New LWP 50233]<br>
09:09:19 [New LWP 50226]<br>
09:09:19 [New LWP 50219]<br>
09:09:19 [New LWP 50216]<br>
09:09:19 [New LWP 50189]<br>
09:09:19 [New LWP 50175]<br>
09:09:19 [New LWP 50161]<br>
09:09:19 [New LWP 50147]<br>
09:09:19 [New LWP 50133]<br>
09:09:19 [New LWP 50119]<br>
09:09:19 [New LWP 50105]<br>
09:09:19 [New LWP 50091]<br>
09:09:19 [New LWP 49889]<br>
09:09:19 [New LWP 49858]<br>
09:09:19 [Thread debugging using libthread_db enabled]<br>
09:09:19 Using host libthread_db library "/lib64/libthread_db.so.1".<br>
09:09:20 0x00007fb5ffece8a3 in select () at<br>
../sysdeps/unix/syscall-template.S:84<br>
09:09:20 84 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)<br>
09:09:20<br>
09:09:20 Thread 34 (Thread 0x7fb5ab7fe700 (LWP 49858)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 33 (Thread 0x7fb5a9899700 (LWP 49889)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 32 (Thread 0x7fb58effd700 (LWP 50091)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2<br>
09:09:20 r = pollster.poll(timeout)<br>
09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop<br>
09:09:20 poll_fun(timeout, map)<br>
09:09:20 File<br>
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py",<br>
line 212, in process_requests<br>
09:09:20 count=1,<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 31 (Thread 0x7fb58e7fc700 (LWP 50105)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2<br>
09:09:20 r = pollster.poll(timeout)<br>
09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop<br>
09:09:20 poll_fun(timeout, map)<br>
09:09:20 File<br>
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py",<br>
line 212, in process_requests<br>
09:09:20 count=1,<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 30 (Thread 0x7fb58dffb700 (LWP 50119)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2<br>
09:09:20 r = pollster.poll(timeout)<br>
09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop<br>
09:09:20 poll_fun(timeout, map)<br>
09:09:20 File<br>
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py",<br>
line 212, in process_requests<br>
09:09:20 count=1,<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 29 (Thread 0x7fb58d7fa700 (LWP 50133)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2<br>
09:09:20 r = pollster.poll(timeout)<br>
09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop<br>
09:09:20 poll_fun(timeout, map)<br>
09:09:20 File<br>
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py",<br>
line 212, in process_requests<br>
09:09:20 count=1,<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 28 (Thread 0x7fb58cff9700 (LWP 50147)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2<br>
09:09:20 r = pollster.poll(timeout)<br>
09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop<br>
09:09:20 poll_fun(timeout, map)<br>
09:09:20 File<br>
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py",<br>
line 212, in process_requests<br>
09:09:20 count=1,<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 27 (Thread 0x7fb56bfff700 (LWP 50161)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2<br>
09:09:20 r = pollster.poll(timeout)<br>
09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop<br>
09:09:20 poll_fun(timeout, map)<br>
09:09:20 File<br>
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py",<br>
line 212, in process_requests<br>
09:09:20 count=1,<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 26 (Thread 0x7fb56b7fe700 (LWP 50175)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2<br>
09:09:20 r = pollster.poll(timeout)<br>
09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop<br>
09:09:20 poll_fun(timeout, map)<br>
09:09:20 File<br>
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py",<br>
line 212, in process_requests<br>
09:09:20 count=1,<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 25 (Thread 0x7fb56affd700 (LWP 50189)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2<br>
09:09:20 r = pollster.poll(timeout)<br>
09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop<br>
09:09:20 poll_fun(timeout, map)<br>
09:09:20 File<br>
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py",<br>
line 212, in process_requests<br>
09:09:20 count=1,<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 24 (Thread 0x7fb59affd700 (LWP 50216)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 23 (Thread 0x7fb5e905f700 (LWP 50219)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 22 (Thread 0x7fb59bfff700 (LWP 50226)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 21 (Thread 0x7fb5e885e700 (LWP 50233)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 20 (Thread 0x7fb5e9860700 (LWP 50240)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 19 (Thread 0x7fb5eb864700 (LWP 50247)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 18 (Thread 0x7fb5c9061700 (LWP 50364)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 17 (Thread 0x7fb5cb7fe700 (LWP 50371)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 16 (Thread 0x7fb5cbfff700 (LWP 50378)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 15 (Thread 0x7fb5ea862700 (LWP 50385)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 14 (Thread 0x7fb5eb063700 (LWP 50392)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 13 (Thread 0x7fb5aaffd700 (LWP 50399)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 12 (Thread 0x7fb5a8f18700 (LWP 50406)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 11 (Thread 0x7fb59b7fe700 (LWP 50413)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 10 (Thread 0x7fb59a7fc700 (LWP 50420)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 9 (Thread 0x7fb599ffb700 (LWP 50427)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 8 (Thread 0x7fb5997fa700 (LWP 50434)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 7 (Thread 0x7fb598ff9700 (LWP 50441)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 6 (Thread 0x7fb58ffff700 (LWP 50448)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 5 (Thread 0x7fb58f7fe700 (LWP 50455)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 4 (Thread 0x7fb56a7fc700 (LWP 50462)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 3 (Thread 0x7fb569ffb700 (LWP 50469)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 2 (Thread 0x7fb5697fa700 (LWP 50476)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in<br>
NoIntrPoll<br>
09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms<br>
09:09:20 File<br>
"/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in<br>
_communicate<br>
09:09:20 pollres = NoIntrPoll(poller.poll, 5)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run<br>
09:09:20 self.__target(*self.__args, **self.__kwargs)<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in<br>
__bootstrap_inner<br>
09:09:20 self.run()<br>
09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap<br>
09:09:20 self.__bootstrap_inner()<br>
09:09:20<br>
09:09:20 Thread 1 (Thread 0x7fb60109d700 (LWP 49417)):<br>
09:09:20 Traceback (most recent call first):<br>
09:09:20 File<br>
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testlibTests.py",<br>
line 264, in test_stuck<br>
09:09:20 time.sleep(300)<br>
09:09:20 File "/usr/lib64/python2.7/unittest/case.py", line 367, in run<br>
09:09:20 testMethod()<br>
09:09:20 File "/usr/lib64/python2.7/unittest/case.py", line 431, in __call__<br>
09:09:20 return self.run(*args, **kwds)<br>
09:09:20 File "/usr/lib/python2.7/site-packages/nose/case.py", line<br>
151, in runTest<br>
09:09:20 test(result)<br>
09:09:20 File "/usr/lib/python2.7/site-packages/nose/case.py", line<br>
133, in run<br>
09:09:20 self.runTest(result)<br>
09:09:20 File "/usr/lib/python2.7/site-packages/nose/case.py", line<br>
45, in __call__<br>
09:09:20 return self.run(*arg, **kwarg)<br>
09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line<br>
224, in run<br>
09:09:20 test(orig)<br>
09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line<br>
177, in __call__<br>
09:09:20 return self.run(*arg, **kw)<br>
09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line<br>
224, in run<br>
09:09:20 test(orig)<br>
09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line<br>
177, in __call__<br>
09:09:20 return self.run(*arg, **kw)<br>
09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line<br>
224, in run<br>
09:09:20 test(orig)<br>
09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line<br>
177, in __call__<br>
09:09:20 return self.run(*arg, **kw)<br>
09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line<br>
224, in run<br>
09:09:20 test(orig)<br>
09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line<br>
177, in __call__<br>
09:09:20 return self.run(*arg, **kw)<br>
09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line 62, in run<br>
09:09:20 test(result)<br>
09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line<br>
207, in runTests<br>
09:09:20 result = self.testRunner.run(self.test)<br>
09:09:20 File "/usr/lib64/python2.7/unittest/main.py", line 95, in __init__<br>
09:09:20 self.runTests()<br>
09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line<br>
121, in __init__<br>
09:09:20 **extra_args)<br>
09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line<br>
301, in run<br>
09:09:20 return TestProgram(*arg, **kw).success<br>
09:09:20 File<br>
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testlib.py",<br>
line 396, in run<br>
09:09:20 sys.exit(not core.run(config=conf, testRunner=runner, argv=argv))<br>
09:09:20 File "../tests/testrunner.py", line 88, in <module><br>
09:09:20 testlib.run()<br>
09:09:20<br>
09:09:20 ========================================================================<br>
09:09:20 = Aborting tests<br>
=<br>
09:09:20 ========================================================================<br>
_______________________________________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/devel</a><br>
</blockquote></div><br></div><div class="gmail_extra">Thank you for pointing this out, in networking there is one direct usage.. taking it out.<br></div><div class="gmail_extra">Thanks,<br></div><div class="gmail_extra">Edy.<br></div></div>