- Devel - oVirt List Archives

[VDSM] stuck tests in the ci - update
by Nir Soffer 02 Jun '16

02 Jun '16

Hi all, So we have this patch, aborting stuck tests and printing a backtrace of all threads: https://gerrit.ovirt.org/#/c/58212 On fedora, we get now a nice backtrace (see bellow) - which show that we have unexpected threads running during the tests. we have 25 iopproces threads like this: 09:09:20 Thread 34 (Thread 0x7fb5ab7fe700 (LWP 49858)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() This is expected given the current code managing these, and are daemon threads. We have 8 reactor threads: 09:09:20 Thread 32 (Thread 0x7fb58effd700 (LWP 50091)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() I don't know where these are started and it they are daemon threads. Searching the tests, we have too much code creating threads directly, instead of using vdsm.concurrent.thread: $ git grep Thread tests/ | wc -l 79 Please check your tests, and replace all threads to use testlib.start_thread() creating daemon threads. Code inheriting from threading.Thread is the worst, please keep a thread instance instead, and start it with testlib.start_thread(). I sent this patch fixing sslTests for now: https://gerrit.ovirt.org/58435 Nir ---- 09:06:05 testlibTests.TestStuckProcess 09:09:19 test_stuck 09:09:19 ======================================================================== 09:09:19 = Timeout completing tests - extracting stacktrace = 09:09:19 ======================================================================== 09:09:19 09:09:19 attach: No such file or directory. 09:09:19 [New LWP 50476] 09:09:19 [New LWP 50469] 09:09:19 [New LWP 50462] 09:09:19 [New LWP 50455] 09:09:19 [New LWP 50448] 09:09:19 [New LWP 50441] 09:09:19 [New LWP 50434] 09:09:19 [New LWP 50427] 09:09:19 [New LWP 50420] 09:09:19 [New LWP 50413] 09:09:19 [New LWP 50406] 09:09:19 [New LWP 50399] 09:09:19 [New LWP 50392] 09:09:19 [New LWP 50385] 09:09:19 [New LWP 50378] 09:09:19 [New LWP 50371] 09:09:19 [New LWP 50364] 09:09:19 [New LWP 50247] 09:09:19 [New LWP 50240] 09:09:19 [New LWP 50233] 09:09:19 [New LWP 50226] 09:09:19 [New LWP 50219] 09:09:19 [New LWP 50216] 09:09:19 [New LWP 50189] 09:09:19 [New LWP 50175] 09:09:19 [New LWP 50161] 09:09:19 [New LWP 50147] 09:09:19 [New LWP 50133] 09:09:19 [New LWP 50119] 09:09:19 [New LWP 50105] 09:09:19 [New LWP 50091] 09:09:19 [New LWP 49889] 09:09:19 [New LWP 49858] 09:09:19 [Thread debugging using libthread_db enabled] 09:09:19 Using host libthread_db library "/lib64/libthread_db.so.1". 09:09:20 0x00007fb5ffece8a3 in select () at ../sysdeps/unix/syscall-template.S:84 09:09:20 84 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) 09:09:20 09:09:20 Thread 34 (Thread 0x7fb5ab7fe700 (LWP 49858)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 33 (Thread 0x7fb5a9899700 (LWP 49889)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 32 (Thread 0x7fb58effd700 (LWP 50091)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 31 (Thread 0x7fb58e7fc700 (LWP 50105)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 30 (Thread 0x7fb58dffb700 (LWP 50119)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 29 (Thread 0x7fb58d7fa700 (LWP 50133)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 28 (Thread 0x7fb58cff9700 (LWP 50147)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 27 (Thread 0x7fb56bfff700 (LWP 50161)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 26 (Thread 0x7fb56b7fe700 (LWP 50175)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 25 (Thread 0x7fb56affd700 (LWP 50189)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 09:09:20 r = pollster.poll(timeout) 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop 09:09:20 poll_fun(timeout, map) 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", line 212, in process_requests 09:09:20 count=1, 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 24 (Thread 0x7fb59affd700 (LWP 50216)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 23 (Thread 0x7fb5e905f700 (LWP 50219)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 22 (Thread 0x7fb59bfff700 (LWP 50226)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 21 (Thread 0x7fb5e885e700 (LWP 50233)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 20 (Thread 0x7fb5e9860700 (LWP 50240)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 19 (Thread 0x7fb5eb864700 (LWP 50247)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 18 (Thread 0x7fb5c9061700 (LWP 50364)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 17 (Thread 0x7fb5cb7fe700 (LWP 50371)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 16 (Thread 0x7fb5cbfff700 (LWP 50378)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 15 (Thread 0x7fb5ea862700 (LWP 50385)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 14 (Thread 0x7fb5eb063700 (LWP 50392)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 13 (Thread 0x7fb5aaffd700 (LWP 50399)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 12 (Thread 0x7fb5a8f18700 (LWP 50406)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 11 (Thread 0x7fb59b7fe700 (LWP 50413)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 10 (Thread 0x7fb59a7fc700 (LWP 50420)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 9 (Thread 0x7fb599ffb700 (LWP 50427)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 8 (Thread 0x7fb5997fa700 (LWP 50434)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 7 (Thread 0x7fb598ff9700 (LWP 50441)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 6 (Thread 0x7fb58ffff700 (LWP 50448)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 5 (Thread 0x7fb58f7fe700 (LWP 50455)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 4 (Thread 0x7fb56a7fc700 (LWP 50462)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 3 (Thread 0x7fb569ffb700 (LWP 50469)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 2 (Thread 0x7fb5697fa700 (LWP 50476)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in NoIntrPoll 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms 09:09:20 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in _communicate 09:09:20 pollres = NoIntrPoll(poller.poll, 5) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run 09:09:20 self.__target(*self.__args, **self.__kwargs) 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 09:09:20 self.run() 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in __bootstrap 09:09:20 self.__bootstrap_inner() 09:09:20 09:09:20 Thread 1 (Thread 0x7fb60109d700 (LWP 49417)): 09:09:20 Traceback (most recent call first): 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testlibTests.py", line 264, in test_stuck 09:09:20 time.sleep(300) 09:09:20 File "/usr/lib64/python2.7/unittest/case.py", line 367, in run 09:09:20 testMethod() 09:09:20 File "/usr/lib64/python2.7/unittest/case.py", line 431, in __call__ 09:09:20 return self.run(*args, **kwds) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest 09:09:20 test(result) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run 09:09:20 self.runTest(result) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/case.py", line 45, in __call__ 09:09:20 return self.run(*arg, **kwarg) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 224, in run 09:09:20 test(orig) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 177, in __call__ 09:09:20 return self.run(*arg, **kw) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 224, in run 09:09:20 test(orig) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 177, in __call__ 09:09:20 return self.run(*arg, **kw) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 224, in run 09:09:20 test(orig) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 177, in __call__ 09:09:20 return self.run(*arg, **kw) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 224, in run 09:09:20 test(orig) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line 177, in __call__ 09:09:20 return self.run(*arg, **kw) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line 62, in run 09:09:20 test(result) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line 207, in runTests 09:09:20 result = self.testRunner.run(self.test) 09:09:20 File "/usr/lib64/python2.7/unittest/main.py", line 95, in __init__ 09:09:20 self.runTests() 09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line 121, in __init__ 09:09:20 **extra_args) 09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line 301, in run 09:09:20 return TestProgram(*arg, **kw).success 09:09:20 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testlib.py", line 396, in run 09:09:20 sys.exit(not core.run(config=conf, testRunner=runner, argv=argv)) 09:09:20 File "../tests/testrunner.py", line 88, in <module> 09:09:20 testlib.run() 09:09:20 09:09:20 ======================================================================== 09:09:20 = Aborting tests = 09:09:20 ========================================================================

2 1

[ACTION REQUESTED] i18n default English text is now stored in properties files
by Scott Dickerson 01 Jun '16

01 Jun '16

Hi, In order to resolve bug [1] and prep [2], the default English text for I18N Constants and Messages have been moved to their corresponding properties files. Going forward, if a new constant or message needs to be added, please add the key method to the proper interface and then add the key plus English text to the interface's corresponding properties file. Checks will be added, in the next few days, to fail the build process if the GWT annotations (@DefaultStringValue, @DefaultMessage) are used. This change has been made to simplify the translation process [1] and to prepare for the replacement of the GWT i18n engine in 4.1 [2]. For example, in the webadmin project, the ApplicationConstants [3] interface previously looked like: package org.ovirt.engine.ui.webadmin; import org.ovirt.engine.ui.common.CommonApplicationConstants; public interface ApplicationConstants extends CommonApplicationConstants { @DefaultStringValue("oVirt Engine Web Administration") String applicationTitle(); @DefaultStringValue("About") String aboutPopupCaption(); @DefaultStringValue("oVirt Engine Version:") String ovirtVersionAbout(); Now, the interface looks like this [4]: package org.ovirt.engine.ui.webadmin; import org.ovirt.engine.ui.common.CommonApplicationConstants; public interface ApplicationConstants extends CommonApplicationConstants { String applicationTitle(); String aboutPopupCaption(); String ovirtVersionAbout(); With a new properties file created, ApplicationContants.properties [5], with this kind of content : # # # Bundle name: org.ovirt.engine.ui.webadmin.ApplicationConstants # # #Thu May 19 01:24:21 EDT 2016 aboutPopupCaption=About applicationTitle=oVirt Engine Web Administration ovirtVersionAbout=oVirt Engine Version\: Let me know if there are any questions. Regards, Scott -- Scott Dickerson Senior Software Engineer RHEV-M Engineering - UX Team Red Hat, Inc [1] - https://bugzilla.redhat.com/show_bug.cgi?id=1224423 [2] - https://bugzilla.redhat.com/show_bug.cgi?id=1287408 [3] - https://github.com/oVirt/ovirt-engine/blob/569695916d2325d0c1e1fdabb6f85cd9… [4] - https://github.com/oVirt/ovirt-engine/blob/master/frontend/webadmin/modules… [5] - https://github.com/oVirt/ovirt-engine/blob/master/frontend/webadmin/modules…

4 6

Interacting with VDSM storage without oVirt Engine
by Hayley Swimelar 01 Jun '16

01 Jun '16

Hello all, Like the title says, I need to work with VDSM's storage layer without involving the engine. Presently, I'm testing this with NFS domains and have been able to use vdsClient to create, attach, and activate a new storage domain, but I have not been able to create a new image or volume. Here are the commands I ran to get to this step, starting with a nfs volume already mounted on the machine I ran the commands on. ---- vdsClient -s host_name createStorageDomain 1 97338d5c-9b6f-1859-b827-e977ed082e53 cli_domain storage_server:/data vdsClient -s host_name attachStorageDomain 97338d5c-9b6f-1859-b827-e977ed082e53 00000001-0001-0001-0001-00000000033c vdsClient -s host_name activateStorageDomain 97338d5c-9b6f-1859-b827-e977ed082e53 00000001-0001-0001-0001-00000000033c vdsClient -s frodo createVolume 97338d5c-9b6f-1859-b827-e977ed082e53 00000001-0001-0001-0001-00000000033c 0288f410-71f1-4b7d-bdb7-e815a93e34ef 53690000000 5 1 2 96d7726c-6969-40a5-84bd-0925496b6051 cli_volume ---- The last command outputs a new UUID, but there is nothing under /rhev/data-center/00000001-0001-0001-0001-00000000033c/97338d5c-9b6f-1859-b827-e977ed082e53/images/ directory. If I pass createVolume an existing image/storage domain UUID, the command still outputs a new UUID, but nothing changes under the images directory. Is there a step that I am missing somewhere in this process? -- Hayley Swimelar LINBIT | Keeping the Digital World Running DRBD — Corosync — Pacemaker +1-503-573-1262 x212

6 12

Empty Dropdown Menus in Webadmin
by Phillip Bailey 01 Jun '16

01 Jun '16

Hi, Is anyone else experiencing empty dropdown menus in webadmin? I started having the problem after rebasing Friday afternoon. I rebased again this morning, but am still experiencing the same problem. I'm trying to determine whether this is something isolated to my environment, or a more widespread issue. I appreciate any advice any of you may be able to offer. Thanks! -Phillip Bailey

3 3

[VDSM] stuck tests in ci
by Nir Soffer 31 May '16

31 May '16

Hi all, I found another stuck build today: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/1151/conso… 11:27:18 ------------------------------------------------------------------------------------------------------------------------------------------------ 11:27:18 TOTAL 40513 21121 48% 11:27:18 ---------------------------------------------------------------------- 11:27:18 Ran 2169 tests in 145.934s 11:27:18 11:27:18 OK (SKIP=88) 11:27:18 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x7fd7c9f2d3d0>> ignored [...] 11:27:18 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x7fd7c9f15550>> ignored 11:27:18 Exception in thread ioprocess communication (6533) (most likely raised during interpreter shutdown): 11:27:18 Traceback (most recent call last): 11:27:18 File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner 11:27:18 File "/usr/lib64/python2.7/threading.py", line 757, in run 11:27:18 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 180, in _communicate 11:27:18 <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'close' This seems smells like a non-daemon thread started by some code, blocking hte test process. I suspect ioprocess, starting such thread, looking into it. Meanwhile, please: - verify that all threads in actual code and in the tests are daemon threads - convert your threads to use vdsm.concurrent.thread instead of threading.Thread (daemon by default) - watch your builds and abort stuck builds David, we need a timeout in the ci, aborting the job after a project based timeout, maybe defined in the project yaml. Cheers, Nir

3 8

[vdsm] Master build fails on f23
by Piotr Kliczewski 31 May '16

31 May '16

vdsm build fails with: 17:18:54 Compiling './vdsm/clientIF.py'... 17:18:54 Makefile:984: recipe for target 'python3' failed 17:18:54 make: *** [python3] Error 1 17:18:54 Took 33 seconds Thanks, Piotr

1 0

oVirt Reports Portal localization
by Juliette Tux 31 May '16

31 May '16

Hello gentlemen, Russian localizators here :) Which files should we look into to be able to translate the Reports Portal interface/messages for oVirt ver 3.x ? Did not find anything relevant on Zanata in case you would like to redirect me over there :) -- С уважением, Дронова Юлия

2 2

Failing MacPoolDaoTests on master
by Roman Mohr 31 May '16

31 May '16

Hi, DAO tests are failing on master in org.ovirt.engine.core.dao.MacPoolDaoTest. Build logs are here: https://travis-ci.org/oVirt/ovirt-engine/builds/134082394 Best Regards, Roman

5 14

dao tests status in CI
by Eyal Edri 31 May '16

31 May '16

FYI, All ovirt-engine dao tests (3.6,4.0,master) were moved to new jenkins and the ones in the old one were disabled. The jobs are still running in "old" mode and not yamelized. Works is being done now to convert them to run in standard CI mode, under mock which should add much more stability to flexibility to them for any developer who would want to change/extend them. -- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

1 0

[vdsm] what happens to ovirmgmt during new host deployment?
by Kai Kang 31 May '16

31 May '16

Hi, I sent mails to ovirt-user maillist for help. But it seems there is no useful info in vdsm.log and supervdsm.log. I want to debug the code and so ask for help here. I am using ovirt 3.6.4.1 and vdsm 4.17.24. When deploy a new host, bridge ovirtmgmt has been created with port eth0. But it fails with: Host dell12 does not comply with the cluster Default networks, the following networks are missing on host: 'ovirtmgmt' I can make ovirtmgmt work by selecting the tab 'Hosts' -> node name(dell12) -> subtab 'Network Interface', click 'Setup Host Networks', drag ovirtmgmt to link with ethernet interface eth0. What engine did to bridge ovirtmgmt during new host deployment? Just call "getVdsCaps" to check whether ovirtmgmt is ready or sent setupNetwork command to configure ovirtmgmt? Which part of vdsm code should I check please? Thanks a lot. --Kai

3 3