Re: [ovirt-devel] [VDSM] stuck tests in ci

The issue is non-daemon thread blocking the python process during shutdown of the tests. Current ioprocess does not create such thread, but we still see this issue today: http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1841/console If the builds are using the latest ioprocess build (0.16.0-1), built after Sun May 15 21:29:24 2016 +0300, this is probably not related to ioprocess To understand this issue we need to get a stacktrace from the stuck python process. See relevant log bellow. Nir ---- 11:49:30 -------------------------------------------------------------------------------------------------------------------------------------------- 11:49:30 TOTAL 40672 21072 48% 11:49:30 ---------------------------------------------------------------------- 11:49:30 Ran 2182 tests in 147.661s 11:49:30 11:49:30 OK (SKIP=94) 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'udev_unref'" in <bound method Context.__del__ of <pyudev.core.Context object at 0x7312610>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x5435350>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x5420850>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x7269910>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x5420f90>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x5419c10>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x610e610>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x6dc4d50>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x610f390>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x54195d0>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x5432510>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x6110250>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x5414750>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x5414150>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x6a33390>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x543d590>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x6ba8390>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x5432d50>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x5435a90>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x503f350>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x5420250>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x5442e90>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x503f950>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x610e7d0>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x5414d90>> ignored 11:49:30 Exception AttributeError: "'NoneType' object has no attribute 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at 0x543d450>> ignored 11:49:30 Exception in thread ioprocess communication (8008) (most likely raised during interpreter shutdown): 11:49:30 Traceback (most recent call last): 11:49:30 File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner 11:49:30 File "/usr/lib64/python2.7/threading.py", line 764, in run 11:49:30 File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 180, in _communicate 11:49:30 <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'close' 17:42:17 Build timed out (after 360 minutes). Marking the build as failed. 17:42:17 Build was aborted On Fri, May 20, 2016 at 10:30 AM, Piotr Kliczewski <piotr.kliczewski(a)gmail.com&gt; wrote: