Re: [ovirt-devel] [VDSM] stuck tests in ci

One more occurrence of the issue [1] [1] http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1359/console On Sun, May 15, 2016 at 8:37 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > The ioprocess issue fixed in https://gerrit.ovirt.org/57473 > > Will be merge soon and available via ovirt-release-master. > > Nir > > On Sun, May 15, 2016 at 7:45 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >> Hi all, >> >> I found another stuck build today: >> http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/1151/con... >> >> 11:27:18 ------------------------------------------------------------------------------------------------------------------------------------------------ >> 11:27:18 TOTAL >> 40513 21121 >> 48% >> 11:27:18 ---------------------------------------------------------------------- >> 11:27:18 Ran 2169 tests in 145.934s >> 11:27:18 >> 11:27:18 OK (SKIP=88) >> 11:27:18 Exception AttributeError: "'NoneType' object has no attribute >> 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess >> object at 0x7fd7c9f2d3d0>> ignored >> [...] >> 11:27:18 Exception AttributeError: "'NoneType' object has no attribute >> 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess >> object at 0x7fd7c9f15550>> ignored >> 11:27:18 Exception in thread ioprocess communication (6533) (most >> likely raised during interpreter shutdown): >> 11:27:18 Traceback (most recent call last): >> 11:27:18 File "/usr/lib64/python2.7/threading.py", line 804, in >> __bootstrap_inner >> 11:27:18 File "/usr/lib64/python2.7/threading.py", line 757, in run >> 11:27:18 File >> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 180, in >> _communicate >> 11:27:18 <type 'exceptions.AttributeError'>: 'NoneType' object has no >> attribute 'close' >> >> This seems smells like a non-daemon thread started by some code, >> blocking hte test process. >> >> I suspect ioprocess, starting such thread, looking into it. >> >> Meanwhile, please: >> - verify that all threads in actual code and in the tests are daemon threads >> - convert your threads to use vdsm.concurrent.thread instead of >> threading.Thread (daemon by default) >> - watch your builds and abort stuck builds >> >> David, we need a timeout in the ci, aborting the job after a project >> based timeout, maybe >> defined in the project yaml. >> >> Cheers, >> Nir > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel