On Mon, Jun 24, 2019 at 11:54 PM Nir Soffer <nsoffer@redhat.com> wrote:

I had this unrelated failure now, never seen it before.

======================================================================
ERROR: test_event_handler (client_test.VdsmClientTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 274, in wrapper
    return f(*args, **kwargs)
  File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/client_test.py", line 220, in test_event_handler
    client.Test.sendEvent()
  File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/client.py", line 294, in _call
    raise TimeoutError(method, kwargs, timeout)
TimeoutError: Request Test.sendEvent with args {} timed out after 3 seconds
Build:
https://jenkins.ovirt.org/job/vdsm_standard-check-patch/6692//artifact/check-patch.tests-py27.fc29.x86_64/mock_logs/script/stdout_stderr.log
Looking at the captured logs, we see this alarming error:
2019-06-24 19:49:58,108 ERROR (Detector thread) [vds.dispatcher] uncaptured python exception,closing channel <yajsonrpc.betterAsyncore.Dispatcher ('::1', 34444, 0, 0) at 0x7f74983a5c20> (<type 'exceptions.AttributeError'>:'NoneType' object has no attribute 'headers'
[/usr/lib64/python2.7/asyncore.py|readwrite|108]
[/usr/lib64/python2.7/asyncore.py|handle_read_event|449]
[vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71]
[vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168]
[vdsm/lib/vdsm/protocoldetector.py|handle_read|129]
[vdsm/lib/yajsonrpc/stompserver.py|handle_socket|412]
[vdsm/lib/vdsm/rpc/bindingjsonrpc.py|add_socket|54]
[vdsm/lib/yajsonrpc/stompserver.py|createListener|379]
[vdsm/lib/yajsonrpc/stompserver.py|StompListener|345]
[vdsm/lib/yajsonrpc/betterAsyncore.py|__init__|47]
[vdsm/lib/yajsonrpc/betterAsyncore.py|switch_implementation|86]
[vdsm/lib/yajsonrpc/stompserver.py|init|363]
[vdsm/lib/vdsm/rpc/bindingjsonrpc.py|_onAccept|57]
[vdsm/lib/yajsonrpc/stomp.py|set_message_handler|635]
[/usr/lib64/python2.7/asyncore.py|handle_read_event|449]
[vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71]
[vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168]
[vdsm/lib/yajsonrpc/stomp.py|handle_read|411]
[vdsm/lib/yajsonrpc/stomp.py|parse|313]
[vdsm/lib/yajsonrpc/stomp.py|_parse_header|249])
(betterAsyncore:179)

Unhandled exception means we did not implement some handle_error, and we got the default error handling which give

very unhelpful tracebacks:

https://github.com/python/cpython/blob/6cbff564f00fbe17b5443f6986a44ce116d32aee/Lib/asyncore.py#L485

So we have 2 issues:

- missing handle_error - fixing this first will help to fix the actual error bellow

- the actual error: 'NoneType' object has no attribute 'headers', from: yajsonrpc/stomp.py line 249

Another failure today:

https://jenkins.ovirt.org/job/vdsm_standard-check-patch/6810//artifact/check-patch.tests-py27.fc29.x86_64/mock_logs/script/stdout_stderr.log

Marcin, I hope you will be able to reproduce this failure with the new parser tests.