June 2016 - Devel - oVirt List Archives

[VDSM] stuck tests in the ci - update
by Nir Soffer 02 Jun '16

02 Jun '16

Hi all, So we have this patch, aborting stuck tests and printing a backtrace of all threads: https://gerrit.ovirt.org/#/c/58212 On fedora, we get now a nice backtrace (see bellow) - which show that we have unexpected threads running during the tests. we have 25 iopproces threads like this: 09:09:20 Thread 34 (Thread 0x7fb5ab7fe700 (LWP 49858)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() This is expected given the current code managing these, and are daemon threads. We have 8 reactor threads: 09:09:20 Thread 32 (Thread 0x7fb58effd700 (LWP 50091)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() I don't know where these are started and it they are daemon threads. Searching the tests, we have too much code creating threads directly, instead of using vdsm.concurrent.thread: $ git grep Thread tests/ | wc -l 79 Please check your tests, and replace all threads to use testlib.start_thread() creating daemon threads. Code inheriting from threading.Thread is the worst, please keep a thread instance instead, and start it with testlib.start_thread(). I sent this patch fixing sslTests for now: https://gerrit.ovirt.org/58435 Nir ---- 09:06:05 testlibTests.TestStuckProcess 09:09:19 test_stuck 09:09:19 ======================================================================== 09:09:19 = Timeout completing tests - extracting stacktrace = 09:09:19 ======================================================================== 09:09:19 09:09:19 attach: No such file or directory. 09:09:19 [New LWP 50476] 09:09:19 [New LWP 50469] 09:09:19 [New LWP 50462] 09:09:19 [New LWP 50455] 09:09:19 [New LWP 50448] 09:09:19 [New LWP 50441] 09:09:19 [New LWP 50434] 09:09:19 [New LWP 50427] 09:09:19 [New LWP 50420] 09:09:19 [New LWP 50413] 09:09:19 [New LWP 50406] 09:09:19 [New LWP 50399] 09:09:19 [New LWP 50392] 09:09:19 [New LWP 50385] 09:09:19 [New LWP 50378] 09:09:19 [New LWP 50371] 09:09:19 [New LWP 50364] 09:09:19 [New LWP 50247] 09:09:19 [New LWP 50240] 09:09:19 [New LWP 50233] 09:09:19 [New LWP 50226] 09:09:19 [New LWP 50219] 09:09:19 [New LWP 50216] 09:09:19 [New LWP 50189] 09:09:19 [New LWP 50175] 09:09:19 [New LWP 50161] 09:09:19 [New LWP 50147] 09:09:19 [New LWP 50133] 09:09:19 [New LWP 50119] 09:09:19 [New LWP 50105] 09:09:19 [New LWP 50091] 09:09:19 [New LWP 49889] 09:09:19 [New LWP 49858] 09:09:19 [Thread debugging using libthread_db enabled] 09:09:19 Using host libthread_db library "/lib64/libthread_db.so.1". 09:09:20 0x00007fb5ffece8a3 in select () at ../sysdeps/unix/syscall-template.S:84 09:09:20 84 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) 09:09:20 09:09:20 Thread 34 (Thread 0x7fb5ab7fe700 (LWP 49858)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 33 (Thread 0x7fb5a9899700 (LWP 49889)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 32 (Thread 0x7fb58effd700 (LWP 50091)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 31 (Thread 0x7fb58e7fc700 (LWP 50105)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 30 (Thread 0x7fb58dffb700 (LWP 50119)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 29 (Thread 0x7fb58d7fa700 (LWP 50133)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 28 (Thread 0x7fb58cff9700 (LWP 50147)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 27 (Thread 0x7fb56bfff700 (LWP 50161)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 26 (Thread 0x7fb56b7fe700 (LWP 50175)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 25 (Thread 0x7fb56affd700 (LWP 50189)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 24 (Thread 0x7fb59affd700 (LWP 50216)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 23 (Thread 0x7fb5e905f700 (LWP 50219)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 22 (Thread 0x7fb59bfff700 (LWP 50226)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 21 (Thread 0x7fb5e885e700 (LWP 50233)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 20 (Thread 0x7fb5e9860700 (LWP 50240)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 19 (Thread 0x7fb5eb864700 (LWP 50247)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 18 (Thread 0x7fb5c9061700 (LWP 50364)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 17 (Thread 0x7fb5cb7fe700 (LWP 50371)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 16 (Thread 0x7fb5cbfff700 (LWP 50378)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 15 (Thread 0x7fb5ea862700 (LWP 50385)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 14 (Thread 0x7fb5eb063700 (LWP 50392)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 13 (Thread 0x7fb5aaffd700 (LWP 50399)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 12 (Thread 0x7fb5a8f18700 (LWP 50406)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 11 (Thread 0x7fb59b7fe700 (LWP 50413)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 10 (Thread 0x7fb59a7fc700 (LWP 50420)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 9 (Thread 0x7fb599ffb700 (LWP 50427)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 8 (Thread 0x7fb5997fa700 (LWP 50434)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 7 (Thread 0x7fb598ff9700 (LWP 50441)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 6 (Thread 0x7fb58ffff700 (LWP 50448)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 5 (Thread 0x7fb58f7fe700 (LWP 50455)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 4 (Thread 0x7fb56a7fc700 (LWP 50462)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 3 (Thread 0x7fb569ffb700 (LWP 50469)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 2 (Thread 0x7fb5697fa700 (LWP 50476)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 1 (Thread 0x7fb60109d700 (LWP 49417)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testlibTests.py", line 264, in test_stuck 09:09:20 time.sleep(300) 09:09:20 File "/usr/lib64/python2.7/unittest/case.py", line 367, in run 09:09:20 testMethod() 09:09:20 File "/usr/lib64/python2.7/unittest/case.py", line 431, in __call__ 09:09:20 return self.run(*args, **kwds) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest 09:09:20 test(result) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run 09:09:20 self.runTest(result) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/case.py", line 45, in __call__ 09:09:20 return self.run(*arg, **kwarg) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 224, in run 09:09:20 test(orig) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 177, in __call__ 09:09:20 return self.run(*arg, **kw) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 224, in run 09:09:20 test(orig) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 177, in __call__ 09:09:20 return self.run(*arg, **kw) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 224, in run 09:09:20 test(orig) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 177, in __call__ 09:09:20 return self.run(*arg, **kw) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 224, in run 09:09:20 test(orig) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 177, in __call__ 09:09:20 return self.run(*arg, **kw) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line 62, in run 09:09:20 test(result) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line 207, in runTests 09:09:20 result = self.testRunner.run(self.test) 09:09:20 File "/usr/lib64/python2.7/unittest/main.py", line 95, in __init__ 09:09:20 self.runTests() 09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line 121, in __init__ 09:09:20 **extra_args) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line 301, in run 09:09:20 return TestProgram(*arg, **kw).success 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testlib.py", line 396, in run 09:09:20 sys.exit(not core.run(config=conf, testRunner=runner, argv=argv)) 09:09:20 File "../tests/testrunner.py", line 88, in <module> 09:09:20 testlib.run() 09:09:20 09:09:20 ======================================================================== 09:09:20 = Aborting tests = 09:09:20 ========================================================================

2 1

[ACTION REQUESTED] i18n default English text is now stored in properties files
by Scott Dickerson 01 Jun '16

01 Jun '16

Hi, In order to resolve bug [1] and prep [2], the default English text for I18N Constants and Messages have been moved to their corresponding properties files. Going forward, if a new constant or message needs to be added, please add the key method to the proper interface and then add the key plus English text to the interface's corresponding properties file. Checks will be added, in the next few days, to fail the build process if the GWT annotations (@DefaultStringValue, @DefaultMessage) are used. This change has been made to simplify the translation process [1] and to prepare for the replacement of the GWT i18n engine in 4.1 [2]. For example, in the webadmin project, the ApplicationConstants [3] interface previously looked like: package org.ovirt.engine.ui.webadmin; import org.ovirt.engine.ui.common.CommonApplicationConstants; public interface ApplicationConstants extends CommonApplicationConstants { @DefaultStringValue("oVirt Engine Web Administration") String applicationTitle(); @DefaultStringValue("About") String aboutPopupCaption(); @DefaultStringValue("oVirt Engine Version:") String ovirtVersionAbout(); Now, the interface looks like this [4]: package org.ovirt.engine.ui.webadmin; import org.ovirt.engine.ui.common.CommonApplicationConstants; public interface ApplicationConstants extends CommonApplicationConstants { String applicationTitle(); String aboutPopupCaption(); String ovirtVersionAbout(); With a new properties file created, ApplicationContants.properties [5], with this kind of content : # # # Bundle name: org.ovirt.engine.ui.webadmin.ApplicationConstants # # #Thu May 19 01:24:21 EDT 2016 aboutPopupCaption=About applicationTitle=oVirt Engine Web Administration ovirtVersionAbout=oVirt Engine Version\: Let me know if there are any questions. Regards, Scott -- Scott Dickerson Senior Software Engineer RHEV-M Engineering - UX Team Red Hat, Inc [1] - https://bugzilla.redhat.com/show_bug.cgi?id=1224423 [2] - https://bugzilla.redhat.com/show_bug.cgi?id=1287408 [3] - https://github.com/oVirt/ovirt-engine/blob/569695916d2325d0c1e1fdabb6f85cd9… [4] - https://github.com/oVirt/ovirt-engine/blob/master/frontend/webadmin/modules… [5] - https://github.com/oVirt/ovirt-engine/blob/master/frontend/webadmin/modules…

4 6

Interacting with VDSM storage without oVirt Engine
by Hayley Swimelar 01 Jun '16

01 Jun '16

Hello all, Like the title says, I need to work with VDSM's storage layer without involving the engine. Presently, I'm testing this with NFS domains and have been able to use vdsClient to create, attach, and activate a new storage domain, but I have not been able to create a new image or volume. Here are the commands I ran to get to this step, starting with a nfs volume already mounted on the machine I ran the commands on. ---- vdsClient -s host_name createStorageDomain 1 97338d5c-9b6f-1859-b827-e977ed082e53 cli_domain storage_server:/data vdsClient -s host_name attachStorageDomain 97338d5c-9b6f-1859-b827-e977ed082e53 00000001-0001-0001-0001-00000000033c vdsClient -s host_name activateStorageDomain 97338d5c-9b6f-1859-b827-e977ed082e53 00000001-0001-0001-0001-00000000033c vdsClient -s frodo createVolume 97338d5c-9b6f-1859-b827-e977ed082e53 00000001-0001-0001-0001-00000000033c 0288f410-71f1-4b7d-bdb7-e815a93e34ef 53690000000 5 1 2 96d7726c-6969-40a5-84bd-0925496b6051 cli_volume ---- The last command outputs a new UUID, but there is nothing under /rhev/data-center/00000001-0001-0001-0001-00000000033c/97338d5c-9b6f-1859-b827-e977ed082e53/images/ directory. If I pass createVolume an existing image/storage domain UUID, the command still outputs a new UUID, but nothing changes under the images directory. Is there a step that I am missing somewhere in this process? -- Hayley Swimelar LINBIT | Keeping the Digital World Running DRBD — Corosync — Pacemaker +1-503-573-1262 x212

6 12

Empty Dropdown Menus in Webadmin
by Phillip Bailey 01 Jun '16

01 Jun '16

Hi, Is anyone else experiencing empty dropdown menus in webadmin? I started having the problem after rebasing Friday afternoon. I rebased again this morning, but am still experiencing the same problem. I'm trying to determine whether this is something isolated to my environment, or a more widespread issue. I appreciate any advice any of you may be able to offer. Thanks! -Phillip Bailey

3 3