[ovirt-devel] [VDSM] stuck tests in the ci - update
Edward Haas
ehaas at redhat.com
Thu Jun 2 07:26:16 UTC 2016
On Wed, Jun 1, 2016 at 1:44 PM, Nir Soffer <nsoffer at redhat.com> wrote:
> Hi all,
>
> So we have this patch, aborting stuck tests and printing a backtrace
> of all threads:
> https://gerrit.ovirt.org/#/c/58212
>
> On fedora, we get now a nice backtrace (see bellow) - which show that
> we have unexpected
> threads running during the tests.
>
> we have 25 iopproces threads like this:
>
> 09:09:20 Thread 34 (Thread 0x7fb5ab7fe700 (LWP 49858)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
>
> This is expected given the current code managing these, and are daemon
> threads.
>
> We have 8 reactor threads:
>
> 09:09:20 Thread 32 (Thread 0x7fb58effd700 (LWP 50091)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2
> 09:09:20 r = pollster.poll(timeout)
> 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop
> 09:09:20 poll_fun(timeout, map)
> 09:09:20 File
>
> "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py",
> line 212, in process_requests
> 09:09:20 count=1,
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
>
> I don't know where these are started and it they are daemon threads.
>
> Searching the tests, we have too much code creating threads directly,
> instead of using vdsm.concurrent.thread:
>
> $ git grep Thread tests/ | wc -l
> 79
>
> Please check your tests, and replace all threads to use
> testlib.start_thread()
> creating daemon threads.
>
> Code inheriting from threading.Thread is the worst, please keep a thread
> instance instead, and start it with testlib.start_thread().
>
> I sent this patch fixing sslTests for now:
> https://gerrit.ovirt.org/58435
>
> Nir
>
> ----
>
> 09:06:05 testlibTests.TestStuckProcess
> 09:09:19 test_stuck
> 09:09:19
> ========================================================================
> 09:09:19 = Timeout completing tests - extracting stacktrace
> =
> 09:09:19
> ========================================================================
> 09:09:19
> 09:09:19 attach: No such file or directory.
> 09:09:19 [New LWP 50476]
> 09:09:19 [New LWP 50469]
> 09:09:19 [New LWP 50462]
> 09:09:19 [New LWP 50455]
> 09:09:19 [New LWP 50448]
> 09:09:19 [New LWP 50441]
> 09:09:19 [New LWP 50434]
> 09:09:19 [New LWP 50427]
> 09:09:19 [New LWP 50420]
> 09:09:19 [New LWP 50413]
> 09:09:19 [New LWP 50406]
> 09:09:19 [New LWP 50399]
> 09:09:19 [New LWP 50392]
> 09:09:19 [New LWP 50385]
> 09:09:19 [New LWP 50378]
> 09:09:19 [New LWP 50371]
> 09:09:19 [New LWP 50364]
> 09:09:19 [New LWP 50247]
> 09:09:19 [New LWP 50240]
> 09:09:19 [New LWP 50233]
> 09:09:19 [New LWP 50226]
> 09:09:19 [New LWP 50219]
> 09:09:19 [New LWP 50216]
> 09:09:19 [New LWP 50189]
> 09:09:19 [New LWP 50175]
> 09:09:19 [New LWP 50161]
> 09:09:19 [New LWP 50147]
> 09:09:19 [New LWP 50133]
> 09:09:19 [New LWP 50119]
> 09:09:19 [New LWP 50105]
> 09:09:19 [New LWP 50091]
> 09:09:19 [New LWP 49889]
> 09:09:19 [New LWP 49858]
> 09:09:19 [Thread debugging using libthread_db enabled]
> 09:09:19 Using host libthread_db library "/lib64/libthread_db.so.1".
> 09:09:20 0x00007fb5ffece8a3 in select () at
> ../sysdeps/unix/syscall-template.S:84
> 09:09:20 84 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> 09:09:20
> 09:09:20 Thread 34 (Thread 0x7fb5ab7fe700 (LWP 49858)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 33 (Thread 0x7fb5a9899700 (LWP 49889)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 32 (Thread 0x7fb58effd700 (LWP 50091)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2
> 09:09:20 r = pollster.poll(timeout)
> 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop
> 09:09:20 poll_fun(timeout, map)
> 09:09:20 File
>
> "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py",
> line 212, in process_requests
> 09:09:20 count=1,
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 31 (Thread 0x7fb58e7fc700 (LWP 50105)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2
> 09:09:20 r = pollster.poll(timeout)
> 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop
> 09:09:20 poll_fun(timeout, map)
> 09:09:20 File
>
> "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py",
> line 212, in process_requests
> 09:09:20 count=1,
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 30 (Thread 0x7fb58dffb700 (LWP 50119)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2
> 09:09:20 r = pollster.poll(timeout)
> 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop
> 09:09:20 poll_fun(timeout, map)
> 09:09:20 File
>
> "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py",
> line 212, in process_requests
> 09:09:20 count=1,
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 29 (Thread 0x7fb58d7fa700 (LWP 50133)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2
> 09:09:20 r = pollster.poll(timeout)
> 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop
> 09:09:20 poll_fun(timeout, map)
> 09:09:20 File
>
> "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py",
> line 212, in process_requests
> 09:09:20 count=1,
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 28 (Thread 0x7fb58cff9700 (LWP 50147)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2
> 09:09:20 r = pollster.poll(timeout)
> 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop
> 09:09:20 poll_fun(timeout, map)
> 09:09:20 File
>
> "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py",
> line 212, in process_requests
> 09:09:20 count=1,
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 27 (Thread 0x7fb56bfff700 (LWP 50161)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2
> 09:09:20 r = pollster.poll(timeout)
> 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop
> 09:09:20 poll_fun(timeout, map)
> 09:09:20 File
>
> "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py",
> line 212, in process_requests
> 09:09:20 count=1,
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 26 (Thread 0x7fb56b7fe700 (LWP 50175)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2
> 09:09:20 r = pollster.poll(timeout)
> 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop
> 09:09:20 poll_fun(timeout, map)
> 09:09:20 File
>
> "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py",
> line 212, in process_requests
> 09:09:20 count=1,
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 25 (Thread 0x7fb56affd700 (LWP 50189)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2
> 09:09:20 r = pollster.poll(timeout)
> 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop
> 09:09:20 poll_fun(timeout, map)
> 09:09:20 File
>
> "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py",
> line 212, in process_requests
> 09:09:20 count=1,
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 24 (Thread 0x7fb59affd700 (LWP 50216)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 23 (Thread 0x7fb5e905f700 (LWP 50219)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 22 (Thread 0x7fb59bfff700 (LWP 50226)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 21 (Thread 0x7fb5e885e700 (LWP 50233)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 20 (Thread 0x7fb5e9860700 (LWP 50240)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 19 (Thread 0x7fb5eb864700 (LWP 50247)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 18 (Thread 0x7fb5c9061700 (LWP 50364)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 17 (Thread 0x7fb5cb7fe700 (LWP 50371)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 16 (Thread 0x7fb5cbfff700 (LWP 50378)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 15 (Thread 0x7fb5ea862700 (LWP 50385)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 14 (Thread 0x7fb5eb063700 (LWP 50392)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 13 (Thread 0x7fb5aaffd700 (LWP 50399)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 12 (Thread 0x7fb5a8f18700 (LWP 50406)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 11 (Thread 0x7fb59b7fe700 (LWP 50413)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 10 (Thread 0x7fb59a7fc700 (LWP 50420)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 9 (Thread 0x7fb599ffb700 (LWP 50427)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 8 (Thread 0x7fb5997fa700 (LWP 50434)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 7 (Thread 0x7fb598ff9700 (LWP 50441)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 6 (Thread 0x7fb58ffff700 (LWP 50448)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 5 (Thread 0x7fb58f7fe700 (LWP 50455)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 4 (Thread 0x7fb56a7fc700 (LWP 50462)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 3 (Thread 0x7fb569ffb700 (LWP 50469)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 2 (Thread 0x7fb5697fa700 (LWP 50476)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in
> NoIntrPoll
> 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms
> 09:09:20 File
> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in
> _communicate
> 09:09:20 pollres = NoIntrPoll(poller.poll, 5)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run
> 09:09:20 self.__target(*self.__args, **self.__kwargs)
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in
> __bootstrap_inner
> 09:09:20 self.run()
> 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in
> __bootstrap
> 09:09:20 self.__bootstrap_inner()
> 09:09:20
> 09:09:20 Thread 1 (Thread 0x7fb60109d700 (LWP 49417)):
> 09:09:20 Traceback (most recent call first):
> 09:09:20 File
>
> "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testlibTests.py",
> line 264, in test_stuck
> 09:09:20 time.sleep(300)
> 09:09:20 File "/usr/lib64/python2.7/unittest/case.py", line 367, in run
> 09:09:20 testMethod()
> 09:09:20 File "/usr/lib64/python2.7/unittest/case.py", line 431, in
> __call__
> 09:09:20 return self.run(*args, **kwds)
> 09:09:20 File "/usr/lib/python2.7/site-packages/nose/case.py", line
> 151, in runTest
> 09:09:20 test(result)
> 09:09:20 File "/usr/lib/python2.7/site-packages/nose/case.py", line
> 133, in run
> 09:09:20 self.runTest(result)
> 09:09:20 File "/usr/lib/python2.7/site-packages/nose/case.py", line
> 45, in __call__
> 09:09:20 return self.run(*arg, **kwarg)
> 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line
> 224, in run
> 09:09:20 test(orig)
> 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line
> 177, in __call__
> 09:09:20 return self.run(*arg, **kw)
> 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line
> 224, in run
> 09:09:20 test(orig)
> 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line
> 177, in __call__
> 09:09:20 return self.run(*arg, **kw)
> 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line
> 224, in run
> 09:09:20 test(orig)
> 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line
> 177, in __call__
> 09:09:20 return self.run(*arg, **kw)
> 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line
> 224, in run
> 09:09:20 test(orig)
> 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line
> 177, in __call__
> 09:09:20 return self.run(*arg, **kw)
> 09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line 62,
> in run
> 09:09:20 test(result)
> 09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line
> 207, in runTests
> 09:09:20 result = self.testRunner.run(self.test)
> 09:09:20 File "/usr/lib64/python2.7/unittest/main.py", line 95, in
> __init__
> 09:09:20 self.runTests()
> 09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line
> 121, in __init__
> 09:09:20 **extra_args)
> 09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line
> 301, in run
> 09:09:20 return TestProgram(*arg, **kw).success
> 09:09:20 File
>
> "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testlib.py",
> line 396, in run
> 09:09:20 sys.exit(not core.run(config=conf, testRunner=runner,
> argv=argv))
> 09:09:20 File "../tests/testrunner.py", line 88, in <module>
> 09:09:20 testlib.run()
> 09:09:20
> 09:09:20
> ========================================================================
> 09:09:20 = Aborting tests
> =
> 09:09:20
> ========================================================================
> _______________________________________________
> Devel mailing list
> Devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
Thank you for pointing this out, in networking there is one direct usage..
taking it out.
Thanks,
Edy.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20160602/14b43a64/attachment-0001.html>
More information about the Devel
mailing list