[ovirt-devel] Re: [VDSM] New random failure on Fedora 29

I had this unrelated failure now, never seen it before. ====================================================================== ERROR: test_event_handler (client_test.VdsmClientTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 274, in wrapper return f(*args, **kwargs) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/client_test.py", line 220, in test_event_handler client.Test.sendEvent() File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/client.py", line 294, in _call raise TimeoutError(method, kwargs, timeout) TimeoutError: Request Test.sendEvent with args {} timed out after 3 seconds Build: https://jenkins.ovirt.org/job/vdsm_standard-check-patch/6692//artifact/ch... Looking at the captured logs, we see this alarming error: 2019-06-24 19:49:58,108 ERROR (Detector thread) [vds.dispatcher] uncaptured python exception,closing channel <yajsonrpc.betterAsyncore.Dispatcher ('::1', 34444, 0, 0) at 0x7f74983a5c20> (<type 'exceptions.AttributeError'>:'NoneType' object has no attribute 'headers' [/usr/lib64/python2.7/asyncore.py|readwrite|108] [/usr/lib64/python2.7/asyncore.py|handle_read_event|449] [vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71] [vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168] [vdsm/lib/vdsm/protocoldetector.py|handle_read|129] [vdsm/lib/yajsonrpc/stompserver.py|handle_socket|412] [vdsm/lib/vdsm/rpc/bindingjsonrpc.py|add_socket|54] [vdsm/lib/yajsonrpc/stompserver.py|createListener|379] [vdsm/lib/yajsonrpc/stompserver.py|StompListener|345] [vdsm/lib/yajsonrpc/betterAsyncore.py|__init__|47] [vdsm/lib/yajsonrpc/betterAsyncore.py|switch_implementation|86] [vdsm/lib/yajsonrpc/stompserver.py|init|363] [vdsm/lib/vdsm/rpc/bindingjsonrpc.py|_onAccept|57] [vdsm/lib/yajsonrpc/stomp.py|set_message_handler|635] [/usr/lib64/python2.7/asyncore.py|handle_read_event|449] [vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71] [vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168] [vdsm/lib/yajsonrpc/stomp.py|handle_read|411] [vdsm/lib/yajsonrpc/stomp.py|parse|313] [vdsm/lib/yajsonrpc/stomp.py|_parse_header|249]) (betterAsyncore:179) Unhandled exception means we did not implement some handle_error, and we got the default error handling which give very unhelpful tracebacks: https://github.com/python/cpython/blob/6cbff564f00fbe17b5443f6986a44ce116... So we have 2 issues: - missing handle_error - fixing this first will help to fix the actual error bellow - the actual error: 'NoneType' object has no attribute 'headers', from: yajsonrpc/stomp.py line 249