[ovirt-devel] [VDSM] stuck tests in ci

[Piotr Kliczewski]

Eyal,

This was ioprocess issue occurring after the fix was provided. I
haven't seen it since build #1389.

Thanks,
Piotr

On Thu, May 19, 2016 at 3:00 PM, Eyal Edri <eedri at redhat.com> wrote:
> was that resolved?
> any infra issue or was it problems with the tests?
>
> On Mon, May 16, 2016 at 3:27 PM, Piotr Kliczewski
> <piotr.kliczewski at gmail.com> wrote:
>>
>> and one more:
>>
>>
>> http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/1389/console
>>
>> On Mon, May 16, 2016 at 1:46 PM, Piotr Kliczewski
>> <piotr.kliczewski at gmail.com> wrote:
>> > One more occurrence of the issue [1]
>> >
>> >
>> > [1]
>> > http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1359/console
>> >
>> > On Sun, May 15, 2016 at 8:37 PM, Nir Soffer <nsoffer at redhat.com> wrote:
>> >> The ioprocess issue fixed in https://gerrit.ovirt.org/57473
>> >>
>> >> Will be merge soon and available via ovirt-release-master.
>> >>
>> >> Nir
>> >>
>> >> On Sun, May 15, 2016 at 7:45 PM, Nir Soffer <nsoffer at redhat.com> wrote:
>> >>> Hi all,
>> >>>
>> >>> I found another stuck build today:
>> >>>
>> >>> http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/1151/console
>> >>>
>> >>> 11:27:18
>> >>> ------------------------------------------------------------------------------------------------------------------------------------------------
>> >>> 11:27:18 TOTAL
>> >>>                                                      40513  21121
>> >>> 48%
>> >>> 11:27:18
>> >>> ----------------------------------------------------------------------
>> >>> 11:27:18 Ran 2169 tests in 145.934s
>> >>> 11:27:18
>> >>> 11:27:18 OK (SKIP=88)
>> >>> 11:27:18 Exception AttributeError: "'NoneType' object has no attribute
>> >>> 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess
>> >>> object at 0x7fd7c9f2d3d0>> ignored
>> >>> [...]
>> >>> 11:27:18 Exception AttributeError: "'NoneType' object has no attribute
>> >>> 'write'" in <bound method IOProcess.__del__ of <ioprocess.IOProcess
>> >>> object at 0x7fd7c9f15550>> ignored
>> >>> 11:27:18 Exception in thread ioprocess communication (6533) (most
>> >>> likely raised during interpreter shutdown):
>> >>> 11:27:18 Traceback (most recent call last):
>> >>> 11:27:18   File "/usr/lib64/python2.7/threading.py", line 804, in
>> >>> __bootstrap_inner
>> >>> 11:27:18   File "/usr/lib64/python2.7/threading.py", line 757, in run
>> >>> 11:27:18   File
>> >>> "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 180, in
>> >>> _communicate
>> >>> 11:27:18 <type 'exceptions.AttributeError'>: 'NoneType' object has no
>> >>> attribute 'close'
>> >>>
>> >>> This seems smells like a non-daemon thread started by some code,
>> >>> blocking hte test process.
>> >>>
>> >>> I suspect ioprocess, starting such thread, looking into it.
>> >>>
>> >>> Meanwhile, please:
>> >>> - verify that all threads in actual code and in the tests are daemon
>> >>> threads
>> >>> - convert your threads to use vdsm.concurrent.thread instead of
>> >>> threading.Thread (daemon by default)
>> >>> - watch your builds and abort stuck builds
>> >>>
>> >>> David, we need a timeout in the ci, aborting the job after a project
>> >>> based timeout, maybe
>> >>> defined in the project yaml.
>> >>>
>> >>> Cheers,
>> >>> Nir
>> >> _______________________________________________
>> >> Devel mailing list
>> >> Devel at ovirt.org
>> >> http://lists.ovirt.org/mailman/listinfo/devel
>> _______________________________________________
>> Devel mailing list
>> Devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>>
>
>
>
> --
> Eyal Edri
> Associate Manager
> RHEV DevOps
> EMEA ENG Virtualization R&D
> Red Hat Israel
>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)