
Hi, On 6/25/19 11:18 PM, Nir Soffer wrote:
On Mon, Jun 24, 2019 at 11:54 PM Nir Soffer <nsoffer@redhat.com <mailto:nsoffer@redhat.com>> wrote:
I had this unrelated failure now, never seen it before. ======================================================================
ERROR: test_event_handler (client_test.VdsmClientTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 274, in wrapper return f(*args, **kwargs) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/client_test.py", line 220, in test_event_handler client.Test.sendEvent() File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/client.py", line 294, in _call raise TimeoutError(method, kwargs, timeout) TimeoutError: Request Test.sendEvent with args {} timed out after 3 seconds
Build: https://jenkins.ovirt.org/job/vdsm_standard-check-patch/6692//artifact/check...
Looking at the captured logs, we see this alarming error:
2019-06-24 19:49:58,108 ERROR (Detector thread) [vds.dispatcher] uncaptured python exception,closing channel <yajsonrpc.betterAsyncore.Dispatcher ('::1', 34444, 0, 0) at 0x7f74983a5c20> (<type 'exceptions.AttributeError'>:'NoneType' object has no attribute 'headers'
[/usr/lib64/python2.7/asyncore.py|readwrite|108] [/usr/lib64/python2.7/asyncore.py|handle_read_event|449] [vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71] [vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168] [vdsm/lib/vdsm/protocoldetector.py|handle_read|129] [vdsm/lib/yajsonrpc/stompserver.py|handle_socket|412] [vdsm/lib/vdsm/rpc/bindingjsonrpc.py|add_socket|54] [vdsm/lib/yajsonrpc/stompserver.py|createListener|379] [vdsm/lib/yajsonrpc/stompserver.py|StompListener|345] [vdsm/lib/yajsonrpc/betterAsyncore.py|__init__|47] [vdsm/lib/yajsonrpc/betterAsyncore.py|switch_implementation|86] [vdsm/lib/yajsonrpc/stompserver.py|init|363] [vdsm/lib/vdsm/rpc/bindingjsonrpc.py|_onAccept|57] [vdsm/lib/yajsonrpc/stomp.py|set_message_handler|635] [/usr/lib64/python2.7/asyncore.py|handle_read_event|449] [vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71] [vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168] [vdsm/lib/yajsonrpc/stomp.py|handle_read|411] [vdsm/lib/yajsonrpc/stomp.py|parse|313] [vdsm/lib/yajsonrpc/stomp.py|_parse_header|249]) (betterAsyncore:179)
Unhandled exception means we did not implement some handle_error, and we got the default error handling which give very unhelpful tracebacks: https://github.com/python/cpython/blob/6cbff564f00fbe17b5443f6986a44ce116d32...
So we have 2 issues: - missing handle_error - fixing this first will help to fix the actual error bellow - the actual error: 'NoneType' object has no attribute 'headers', from: yajsonrpc/stomp.py line 249
Another failure today: https://jenkins.ovirt.org/job/vdsm_standard-check-patch/6810//artifact/check...
Marcin, I hope you will be able to reproduce this failure with the new parser tests. Parser tests won't help here - this is our broken asyncore implementation. An overhaul of asyncore stuff is on my radar, but first I need to work on making basic yajsonrpc/stomp communication scenarios work on py3. If the failure repeats often, I can mark the test as broken on CI for now.
Nir