3:15 a.m.
Hi all,
We see random failures in the stomp tests for long time.
I suggest we mark these tests as broken for now.
See examples at the end of the message.
Nir
----
http://jenkins.phx.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/93/c...
17:53:09 ======================================================================
17:53:09 ERROR: test_event(False) (stompTests.StompTests)
17:53:09 ----------------------------------------------------------------------
17:53:09 Traceback (most recent call last):
17:53:09 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testlib.py",
line 74, in wrapper
17:53:09 return f(self, *args)
17:53:09 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/stompTests.py",
line 110, in test_event
17:53:09 client.callMethod("event", [], str(uuid4()))
17:53:09 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/__init__.py",
line 340, in callMethod
17:53:09 raise JsonRpcNoResponseError(methodName)
17:53:09 JsonRpcNoResponseError: [-32605] No response for JSON-RPC
event request.
17:53:09 -------------------- >> begin captured logging <<
--------------------
17:53:09 vds.MultiProtocolAcceptor: INFO: Listening at ::1:36333
17:53:09 Scheduler: DEBUG: Starting scheduler test.Scheduler
17:53:09 Scheduler: DEBUG: started
17:53:09 Executor: DEBUG: Starting executor
17:53:09 Executor: DEBUG: Starting worker jsonrpc.Executor/0
17:53:09 Executor: DEBUG: Worker started
17:53:09 Executor: DEBUG: Starting worker jsonrpc.Executor/1
17:53:09 Executor: DEBUG: Starting worker jsonrpc.Executor/2
17:53:09 Executor: DEBUG: Worker started
17:53:09 Executor: DEBUG: Starting worker jsonrpc.Executor/3
17:53:09 Executor: DEBUG: Worker started
17:53:09 Executor: DEBUG: Worker started
17:53:09 Executor: DEBUG: Starting worker jsonrpc.Executor/4
17:53:09 Executor: DEBUG: Worker started
17:53:09 Executor: DEBUG: Starting worker jsonrpc.Executor/5
17:53:09 Executor: DEBUG: Worker started
17:53:09 Executor: DEBUG: Starting worker jsonrpc.Executor/6
17:53:09 Executor: DEBUG: Worker started
17:53:09 Executor: DEBUG: Starting worker jsonrpc.Executor/7
17:53:09 Executor: DEBUG: Worker started
17:53:09 FakeClientIf: INFO: XMLRPC server running
17:53:09 vds.MultiProtocolAcceptor: DEBUG: Adding detector
<vdsm.rpc.bindingxmlrpc.XmlDetector instance at 0x7f0191899488>
17:53:09 vds.MultiProtocolAcceptor: DEBUG: Adding detector
<yajsonrpc.stompreactor.StompDetector instance at 0x7f01918994d0>
17:53:09 ProtocolDetector.AcceptorImpl: INFO: Accepting connection
from ::1:39823
17:53:09 ProtocolDetector.Detector: DEBUG: Using required_size=11
17:53:09 ProtocolDetector.Detector: INFO: Detected protocol stomp from ::1:39823
17:53:09 jsonrpc.AsyncoreClient: DEBUG: Sending response
17:53:09 Broker.StompAdapter: INFO: Processing CONNECT request
17:53:09 yajsonrpc.protocols.stomp.AsyncClient: DEBUG: Stomp
connection established
17:53:09 jsonrpc.JsonRpcServer: DEBUG: Calling 'event' in bridge with []
17:53:09 jsonrpc.Notification: DEBUG: Sending event {"params":
{"content": true, "notify_time": 6398709080}, "jsonrpc":
"2.0",
"method": "vdsm.event"}
17:53:09 jsonrpc.JsonRpcServer: DEBUG: Return 'event' in bridge with True
17:53:09 Broker.StompAdapter: INFO: Subscribe command received
17:53:09 protocoldetector.StompDetector: DEBUG: Stomp detected from
('::1', 39823)
17:53:09 vds.MultiProtocolAcceptor: DEBUG: Stopping Acceptor
17:53:09 jsonrpc.JsonRpcServer: INFO: Stopping JsonRPC Server
17:53:09 Executor: DEBUG: Stopping executor
17:53:09 Executor: DEBUG: Worker stopped
17:53:09 Executor: DEBUG: Worker stopped
17:53:09 Executor: DEBUG: Worker stopped
17:53:09 Executor: DEBUG: Worker stopped
17:53:09 Executor: DEBUG: Worker stopped
17:53:09 Executor: DEBUG: Worker stopped
17:53:09 Executor: DEBUG: Worker stopped
17:53:09 Executor: DEBUG: Worker stopped
17:53:09 Executor: DEBUG: Waiting for worker jsonrpc.Executor/0
17:53:09 Executor: DEBUG: Waiting for worker jsonrpc.Executor/7
17:53:09 Executor: DEBUG: Waiting for worker jsonrpc.Executor/2
17:53:09 Executor: DEBUG: Waiting for worker jsonrpc.Executor/1
17:53:09 Executor: DEBUG: Waiting for worker jsonrpc.Executor/5
17:53:09 Executor: DEBUG: Waiting for worker jsonrpc.Executor/3
17:53:09 Executor: DEBUG: Waiting for worker jsonrpc.Executor/6
17:53:09 Executor: DEBUG: Waiting for worker jsonrpc.Executor/4
17:53:09 FakeClientIf: INFO: Stopping XMLRPC server
17:53:09 FakeClientIf: INFO: XMLRPC server stopped
17:53:09 Scheduler: DEBUG: Stopping scheduler test.Scheduler
17:53:09 Scheduler: DEBUG: stopped
17:53:09 --------------------- >> end captured logging <<
---------------------
http://jenkins.phx.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/87/c...
17:14:24 ======================================================================
17:14:24 ERROR: test_event(False) (stompTests.StompTests)
17:14:24 ----------------------------------------------------------------------
17:14:24 Traceback (most recent call last):
17:14:24 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testlib.py",
line 74, in wrapper
17:14:24 return f(self, *args)
17:14:24 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/stompTests.py",
line 110, in test_event
17:14:24 client.callMethod("event", [], str(uuid4()))
17:14:24 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/__init__.py",
line 340, in callMethod
17:14:24 raise JsonRpcNoResponseError(methodName)
17:14:24 JsonRpcNoResponseError: [-32605] No response for JSON-RPC
event request.
17:14:24 -------------------- >> begin captured logging <<
--------------------
17:14:24 vds.MultiProtocolAcceptor: INFO: Listening at ::1:39663
17:14:24 Scheduler: DEBUG: Starting scheduler test.Scheduler
17:14:24 Scheduler: DEBUG: started
17:14:24 Executor: DEBUG: Starting executor
17:14:24 Executor: DEBUG: Starting worker jsonrpc.Executor/0
17:14:24 Executor: DEBUG: Worker started
17:14:24 Executor: DEBUG: Starting worker jsonrpc.Executor/1
17:14:24 Executor: DEBUG: Worker started
17:14:24 Executor: DEBUG: Starting worker jsonrpc.Executor/2
17:14:24 Executor: DEBUG: Starting worker jsonrpc.Executor/3
17:14:24 Executor: DEBUG: Worker started
17:14:24 Executor: DEBUG: Worker started
17:14:24 Executor: DEBUG: Starting worker jsonrpc.Executor/4
17:14:24 Executor: DEBUG: Worker started
17:14:24 Executor: DEBUG: Starting worker jsonrpc.Executor/5
17:14:24 Executor: DEBUG: Worker started
17:14:24 Executor: DEBUG: Starting worker jsonrpc.Executor/6
17:14:24 Executor: DEBUG: Worker started
17:14:24 Executor: DEBUG: Starting worker jsonrpc.Executor/7
17:14:24 Executor: DEBUG: Worker started
17:14:24 FakeClientIf: INFO: XMLRPC server running
17:14:24 vds.MultiProtocolAcceptor: DEBUG: Adding detector
<vdsm.rpc.bindingxmlrpc.XmlDetector instance at 0x7fa3a183c6c8>
17:14:24 vds.MultiProtocolAcceptor: DEBUG: Adding detector
<yajsonrpc.stompreactor.StompDetector instance at 0x7fa3a183c710>
17:14:24 ProtocolDetector.AcceptorImpl: INFO: Accepting connection
from ::1:34641
17:14:24 ProtocolDetector.Detector: DEBUG: Using required_size=11
17:14:24 ProtocolDetector.Detector: INFO: Detected protocol stomp from ::1:34641
17:14:24 jsonrpc.AsyncoreClient: DEBUG: Sending response
17:14:24 Broker.StompAdapter: INFO: Processing CONNECT request
17:14:24 yajsonrpc.protocols.stomp.AsyncClient: DEBUG: Stomp
connection established
17:14:24 jsonrpc.JsonRpcServer: DEBUG: Calling 'event' in bridge with []
17:14:24 jsonrpc.Notification: DEBUG: Sending event {"params":
{"content": true, "notify_time": 11313399420}, "jsonrpc":
"2.0",
"method": "vdsm.event"}
17:14:24 jsonrpc.JsonRpcServer: DEBUG: Return 'event' in bridge with True
17:14:24 Broker.StompAdapter: INFO: Subscribe command received
17:14:24 protocoldetector.StompDetector: DEBUG: Stomp detected from
('::1', 34641)
17:14:24 vds.MultiProtocolAcceptor: DEBUG: Stopping Acceptor
17:14:24 jsonrpc.JsonRpcServer: INFO: Stopping JsonRPC Server
17:14:24 Executor: DEBUG: Stopping executor
17:14:24 Executor: DEBUG: Worker stopped
17:14:24 Executor: DEBUG: Worker stopped
17:14:24 Executor: DEBUG: Worker stopped
17:14:24 Executor: DEBUG: Worker stopped
17:14:24 Executor: DEBUG: Worker stopped
17:14:24 Executor: DEBUG: Worker stopped
17:14:24 Executor: DEBUG: Worker stopped
17:14:24 Executor: DEBUG: Worker stopped
17:14:24 Executor: DEBUG: Waiting for worker jsonrpc.Executor/5
17:14:24 Executor: DEBUG: Waiting for worker jsonrpc.Executor/3
17:14:24 Executor: DEBUG: Waiting for worker jsonrpc.Executor/0
17:14:24 Executor: DEBUG: Waiting for worker jsonrpc.Executor/2
17:14:24 Executor: DEBUG: Waiting for worker jsonrpc.Executor/1
17:14:24 Executor: DEBUG: Waiting for worker jsonrpc.Executor/4
17:14:24 Executor: DEBUG: Waiting for worker jsonrpc.Executor/7
17:14:24 Executor: DEBUG: Waiting for worker jsonrpc.Executor/6
17:14:24 FakeClientIf: INFO: Stopping XMLRPC server
17:14:24 FakeClientIf: INFO: XMLRPC server stopped
17:14:24 Scheduler: DEBUG: Stopping scheduler test.Scheduler
17:14:24 Scheduler: DEBUG: stopped
17:14:24 --------------------- >> end captured logging <<
---------------------
http://jenkins.phx.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/82/c...
17:05:43 ======================================================================
17:05:43 ERROR: test_event(False) (stompTests.StompTests)
17:05:43 ----------------------------------------------------------------------
17:05:43 Traceback (most recent call last):
17:05:43 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testlib.py",
line 74, in wrapper
17:05:43 return f(self, *args)
17:05:43 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/stompTests.py",
line 110, in test_event
17:05:43 client.callMethod("event", [], str(uuid4()))
17:05:43 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/__init__.py",
line 340, in callMethod
17:05:43 raise JsonRpcNoResponseError(methodName)
17:05:43 JsonRpcNoResponseError: [-32605] No response for JSON-RPC
event request.
17:05:43 -------------------- >> begin captured logging <<
--------------------
17:05:43 Scheduler: DEBUG: stopped
17:05:43 vds.MultiProtocolAcceptor: INFO: Listening at ::1:37245
17:05:43 Scheduler: DEBUG: Starting scheduler test.Scheduler
17:05:43 Scheduler: DEBUG: started
17:05:43 Executor: DEBUG: Starting executor
17:05:43 Executor: DEBUG: Starting worker jsonrpc.Executor/0
17:05:43 Executor: DEBUG: Worker started
17:05:43 Executor: DEBUG: Starting worker jsonrpc.Executor/1
17:05:43 Executor: DEBUG: Worker started
17:05:43 Executor: DEBUG: Starting worker jsonrpc.Executor/2
17:05:43 Executor: DEBUG: Worker started
17:05:43 Executor: DEBUG: Starting worker jsonrpc.Executor/3
17:05:43 Executor: DEBUG: Worker started
17:05:43 Executor: DEBUG: Starting worker jsonrpc.Executor/4
17:05:43 Executor: DEBUG: Worker started
17:05:43 Executor: DEBUG: Starting worker jsonrpc.Executor/5
17:05:43 Executor: DEBUG: Worker started
17:05:43 Executor: DEBUG: Starting worker jsonrpc.Executor/6
17:05:43 Executor: DEBUG: Starting worker jsonrpc.Executor/7
17:05:43 Executor: DEBUG: Worker started
17:05:43 Executor: DEBUG: Worker started
17:05:43 FakeClientIf: INFO: XMLRPC server running
17:05:43 vds.MultiProtocolAcceptor: DEBUG: Adding detector
<vdsm.rpc.bindingxmlrpc.XmlDetector instance at 0x7f27fb0d70e0>
17:05:43 vds.MultiProtocolAcceptor: DEBUG: Adding detector
<yajsonrpc.stompreactor.StompDetector instance at 0x7f27fb0e2488>
17:05:43 ProtocolDetector.AcceptorImpl: INFO: Accepting connection
from ::1:52346
17:05:43 ProtocolDetector.Detector: DEBUG: Using required_size=11
17:05:43 jsonrpc.AsyncoreClient: DEBUG: Sending response
17:05:43 ProtocolDetector.Detector: INFO: Detected protocol stomp from ::1:52346
17:05:43 Broker.StompAdapter: INFO: Processing CONNECT request
17:05:43 yajsonrpc.protocols.stomp.AsyncClient: DEBUG: Stomp
connection established
17:05:43 jsonrpc.JsonRpcServer: DEBUG: Calling 'event' in bridge with []
17:05:43 jsonrpc.Notification: DEBUG: Sending event {"params":
{"content": true, "notify_time": 6397426400}, "jsonrpc":
"2.0",
"method": "vdsm.event"}
17:05:43 jsonrpc.JsonRpcServer: DEBUG: Return 'event' in bridge with True
17:05:43 Broker.StompAdapter: INFO: Subscribe command received
17:05:43 protocoldetector.StompDetector: DEBUG: Stomp detected from
('::1', 52346)
17:05:43 vds.MultiProtocolAcceptor: DEBUG: Stopping Acceptor
17:05:43 jsonrpc.JsonRpcServer: INFO: Stopping JsonRPC Server
17:05:43 Executor: DEBUG: Stopping executor
17:05:43 Executor: DEBUG: Worker stopped
17:05:43 Executor: DEBUG: Worker stopped
17:05:43 Executor: DEBUG: Worker stopped
17:05:43 Executor: DEBUG: Worker stopped
17:05:43 Executor: DEBUG: Worker stopped
17:05:43 Executor: DEBUG: Worker stopped
17:05:43 Executor: DEBUG: Worker stopped
17:05:43 Executor: DEBUG: Waiting for worker jsonrpc.Executor/6
17:05:43 Executor: DEBUG: Worker stopped
17:05:43 Executor: DEBUG: Waiting for worker jsonrpc.Executor/3
17:05:43 Executor: DEBUG: Waiting for worker jsonrpc.Executor/1
17:05:43 Executor: DEBUG: Waiting for worker jsonrpc.Executor/4
17:05:43 Executor: DEBUG: Waiting for worker jsonrpc.Executor/0
17:05:43 Executor: DEBUG: Waiting for worker jsonrpc.Executor/5
17:05:43 Executor: DEBUG: Waiting for worker jsonrpc.Executor/2
17:05:43 Executor: DEBUG: Waiting for worker jsonrpc.Executor/7
17:05:43 FakeClientIf: INFO: Stopping XMLRPC server
17:05:43 FakeClientIf: INFO: XMLRPC server stopped
17:05:43 Scheduler: DEBUG: Stopping scheduler test.Scheduler
17:05:43 Scheduler: DEBUG: stopped
17:05:43 --------------------- >> end captured logging <<
---------------------
http://jenkins.phx.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/72/c...
16:32:00 ======================================================================
16:32:00 ERROR: test_echo(4096, False) (stompTests.StompTests)
16:32:00 ----------------------------------------------------------------------
16:32:00 Traceback (most recent call last):
16:32:00 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testlib.py",
line 74, in wrapper
16:32:00 return f(self, *args)
16:32:00 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/stompTests.py",
line 90, in test_echo
16:32:00 str(uuid4())),
16:32:00 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/__init__.py",
line 340, in callMethod
16:32:00 raise JsonRpcNoResponseError(methodName)
16:32:00 JsonRpcNoResponseError: [-32605] No response for JSON-RPC echo request.
16:32:00 -------------------- >> begin captured logging <<
--------------------
16:32:00 vds.MultiProtocolAcceptor: INFO: Listening at ::1:59025
16:32:00 Scheduler: DEBUG: Starting scheduler test.Scheduler
16:32:00 Scheduler: DEBUG: started
16:32:00 Executor: DEBUG: Starting executor
16:32:00 Executor: DEBUG: Starting worker jsonrpc.Executor/0
16:32:00 Executor: DEBUG: Worker started
16:32:00 Executor: DEBUG: Starting worker jsonrpc.Executor/1
16:32:00 Executor: DEBUG: Worker started
16:32:00 Executor: DEBUG: Starting worker jsonrpc.Executor/2
16:32:00 Executor: DEBUG: Worker started
16:32:00 Executor: DEBUG: Starting worker jsonrpc.Executor/3
16:32:00 Executor: DEBUG: Starting worker jsonrpc.Executor/4
16:32:00 Executor: DEBUG: Worker started
16:32:00 Executor: DEBUG: Worker started
16:32:00 Executor: DEBUG: Starting worker jsonrpc.Executor/5
16:32:00 Executor: DEBUG: Worker started
16:32:00 Executor: DEBUG: Starting worker jsonrpc.Executor/6
16:32:00 Executor: DEBUG: Starting worker jsonrpc.Executor/7
16:32:00 Executor: DEBUG: Worker started
16:32:00 Executor: DEBUG: Worker started
16:32:00 FakeClientIf: INFO: XMLRPC server running
16:32:00 vds.MultiProtocolAcceptor: DEBUG: Adding detector
<vdsm.rpc.bindingxmlrpc.XmlDetector instance at 0x7f411903ecf8>
16:32:00 vds.MultiProtocolAcceptor: DEBUG: Adding detector
<yajsonrpc.stompreactor.StompDetector instance at 0x7f411903ed40>
16:32:00 ProtocolDetector.AcceptorImpl: INFO: Accepting connection
from ::1:49920
16:32:00 ProtocolDetector.Detector: DEBUG: Using required_size=11
16:32:00 ProtocolDetector.Detector: INFO: Detected protocol stomp from ::1:49920
16:32:00 Broker.StompAdapter: INFO: Processing CONNECT request
16:32:00 jsonrpc.AsyncoreClient: DEBUG: Sending response
16:32:00 Broker.StompAdapter: INFO: Subscribe command received
16:32:00 vds.dispatcher: ERROR: uncaptured python exception, closing
channel <yajsonrpc.betterAsyncore.Dispatcher ('::1', 49920, 0, 0) at
0x7f411903ef38> (<type 'exceptions.AttributeError'>:'NoneType'
object
has no attribute 'command'
[/usr/lib64/python2.7/asyncore.py|readwrite|108]
[/usr/lib64/python2.7/asyncore.py|handle_read_event|449]
[/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|63]
[/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|137]
[/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/vdsm/protocoldetector.py|handle_read|125]
[/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/stompreactor.py|handle_socket|481]
[/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/vdsm/rpc/bindingjsonrpc.py|add_socket|48]
[/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/stompreactor.py|createListener|449]
[/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/stompreactor.py|StompListener|415]
[/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py|__init__|39]
[/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py|switch_implementation|72]
[/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/stompreactor.py|init|433]
[/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/vdsm/rpc/bindingjsonrpc.py|_onAccept|51]
[/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/stompreactor.py|set_message_handler|297]
[/usr/lib64/python2.7/asyncore.py|handle_read_event|449]
[/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|63]
[/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|137]
[/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/stomp.py|handle_read|367]
[/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/stompreactor.py|handle_frame|260])
16:32:00 vds.dispatcher: WARNING: unhandled close event
16:32:00 jsonrpc.JsonRpcServer: DEBUG: Calling 'echo' in bridge with
[u'Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do
eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
minim veniam, quis nostrud exercitation ullamco laboris nisi ut
aliquip ex ea commodo consequat. Duis aute irure dolor in
reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla
pariatur. Excepteur sint occaecat cupidatat non proident, sunt in
culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum
dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor
incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea
commodo consequat. Duis aute irure dolor in reprehenderit in voluptate
velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint
occaecat cupidatat non proident, sunt in culpa qui officia deserunt
mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur
adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore
magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation
ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute
irure dolor in reprehenderit in voluptate velit esse cillum dolore eu
fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident,
sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem
ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim
veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex
ea commodo consequat. Duis aute irure dolor in reprehenderit in
voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur
sint occaecat cupidatat non proident, sunt in culpa qui officia
deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet,
consectetur adipisicing elit, sed do eiusmod tempor incididunt ut
labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum
dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est
laborum. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed
do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim
ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut
aliquip ex ea commodo consequat. Duis aute irure dolor in
reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla
pariatur. Excepteur sint occaecat cupidatat non proident, sunt in
culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum
dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor
incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea
commodo consequat. Duis aute irure dolor in reprehenderit in voluptate
velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint
occaecat cupidatat non proident, sunt in culpa qui officia deserunt
mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur
adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore
magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation
ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute
irure dolor in reprehenderit in voluptate velit esse cillum dolore eu
fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident,
sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem
ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim
veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex
ea commodo consequat. Duis aute irure dolor in reprehenderit in
voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur
sint occaecat cupidatat non proident, sunt in culpa qui officia
deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet,
consectetur adipisicing elit, sed do eiusmod ']
16:32:01 jsonrpc.JsonRpcServer: DEBUG: Return 'echo' in bridge with
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do
eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
minim veniam, quis nostrud exercitation ullamco laboris nisi ut
aliquip ex ea commodo consequat. Duis aute irure dolor in
reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla
pariatur. Excepteur sint occaecat cupidatat non proident, sunt in
culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum
dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor
incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea
commodo consequat. Duis aute irure dolor in reprehenderit in voluptate
velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint
occaecat cupidatat non proident, sunt in culpa qui officia deserunt
mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur
adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore
magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation
ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute
irure dolor in reprehenderit in voluptate velit esse cillum dolore eu
fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident,
sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem
ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim
veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex
ea commodo consequat. Duis aute irure dolor in reprehenderit in
voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur
sint occaecat cupidatat non proident, sunt in culpa qui officia
deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet,
consectetur adipisicing elit, sed do eiusmod tempor incididunt ut
labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum
dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est
laborum. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed
do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim
ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut
aliquip ex ea commodo consequat. Duis aute irure dolor in
reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla
pariatur. Excepteur sint occaecat cupidatat non proident, sunt in
culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum
dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor
incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea
commodo consequat. Duis aute irure dolor in reprehenderit in voluptate
velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint
occaecat cupidatat non proident, sunt in culpa qui officia deserunt
mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur
adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore
magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation
ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute
irure dolor in reprehenderit in voluptate velit esse cillum dolore eu
fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident,
sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem
ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim
veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex
ea commodo consequat. Duis aute irure dolor in reprehenderit in
voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur
sint occaecat cupidatat non proident, sunt in culpa qui officia
deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet,
consectetur adipisicing elit, sed do eiusmod
16:32:01 yajsonrpc.protocols.stomp.AsyncClient: DEBUG: Stomp
connection established
16:32:01 vds.MultiProtocolAcceptor: DEBUG: Stopping Acceptor
16:32:01 jsonrpc.JsonRpcServer: INFO: Stopping JsonRPC Server
16:32:01 Executor: DEBUG: Stopping executor
16:32:01 Executor: DEBUG: Worker stopped
16:32:01 Executor: DEBUG: Worker stopped
16:32:01 Executor: DEBUG: Worker stopped
16:32:01 Executor: DEBUG: Worker stopped
16:32:01 Executor: DEBUG: Worker stopped
16:32:01 Executor: DEBUG: Worker stopped
16:32:01 Executor: DEBUG: Worker stopped
16:32:01 Executor: DEBUG: Worker stopped
16:32:01 Executor: DEBUG: Waiting for worker jsonrpc.Executor/5
16:32:01 Executor: DEBUG: Waiting for worker jsonrpc.Executor/6
16:32:01 Executor: DEBUG: Waiting for worker jsonrpc.Executor/4
16:32:01 Executor: DEBUG: Waiting for worker jsonrpc.Executor/0
16:32:01 Executor: DEBUG: Waiting for worker jsonrpc.Executor/2
16:32:01 Executor: DEBUG: Waiting for worker jsonrpc.Executor/1
16:32:01 Executor: DEBUG: Waiting for worker jsonrpc.Executor/7
16:32:01 Executor: DEBUG: Waiting for worker jsonrpc.Executor/3
16:32:01 FakeClientIf: INFO: Stopping XMLRPC server
16:32:01 FakeClientIf: INFO: XMLRPC server stopped
16:32:01 Scheduler: DEBUG: Stopping scheduler test.Scheduler
16:32:01 Scheduler: DEBUG: stopped
16:32:01 --------------------- >> end captured logging <<
---------------------
5:17 p.m.
2016-04-25 20:15 GMT+02:00 Nir Soffer <nsoffer(a)redhat.com>:
Hi all,
We see random failures in the stomp tests for long time.
I suggest we mark these tests as broken for now.
See examples at the end of the message.
Nir
----
17:14:24
======================================================================
17:14:24 ERROR: test_event(False) (stompTests.StompTests)
17:14:24 ----------------------------------------------------------------------
17:14:24 Traceback (most recent call last):
17:14:24 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testlib.py",
line 74, in wrapper
17:14:24 return f(self, *args)
17:14:24 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/stompTests.py",
line 110, in test_event
17:14:24 client.callMethod("event", [], str(uuid4()))
17:14:24 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/__init__.py",
line 340, in callMethod
17:14:24 raise JsonRpcNoResponseError(methodName)
17:14:24 JsonRpcNoResponseError: [-32605] No response for JSON-RPC
event request.
I'd suggest to investigate this a bit more since it can hide a serious issue.
I'm moving hosted-engine-setup from XMLRPC to JsonRPC and I'm facing
exactly this kind of issue: it seams that some requests got lost and I
just receive a JsonRpcNoResponseError after a long time.
The real issue is that my request never reached VDSM getting lost
somehow in the queuing mechanism.
5:40 p.m.
Simone,
The issue you are seeing is very interesting - a message we add to a deque
disappear next time we check and according the log you provided there is no
code accessing the deque. It happens only for one specific message. All
the other messages work ok. Can you please gather logs so we could see what
is really happening with it?
Thanks,
Piotr
26 kwi 2016 10:18 "Simone Tiraboschi" <stirabos(a)redhat.com> napisał(a):
2016-04-25 20:15 GMT+02:00 Nir Soffer <nsoffer(a)redhat.com>:
> Hi all,
>
> We see random failures in the stomp tests for long time.
>
> I suggest we mark these tests as broken for now.
>
> See examples at the end of the message.
>
> Nir
>
> ----
>
> 17:14:24
======================================================================
> 17:14:24 ERROR: test_event(False) (stompTests.StompTests)
> 17:14:24
----------------------------------------------------------------------
> 17:14:24 Traceback (most recent call last):
> 17:14:24 File
>
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testlib.py",
> line 74, in wrapper
> 17:14:24 return f(self, *args)
> 17:14:24 File
>
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/stompTests.py",
> line 110, in test_event
> 17:14:24 client.callMethod("event", [], str(uuid4()))
> 17:14:24 File
>
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/__init__.py",
> line 340, in callMethod
> 17:14:24 raise JsonRpcNoResponseError(methodName)
> 17:14:24 JsonRpcNoResponseError: [-32605] No response for JSON-RPC
> event request.
I'd suggest to investigate this a bit more since it can hide a serious
issue.
I'm moving hosted-engine-setup from XMLRPC to JsonRPC and I'm facing
exactly this kind of issue: it seams that some requests got lost and I
just receive a JsonRpcNoResponseError after a long time.
The real issue is that my request never reached VDSM getting lost
somehow in the queuing mechanism.
4:03 p.m.
On Tue, Apr 26, 2016 at 10:40 AM, Piotr Kliczewski <pkliczew(a)redhat.com>
wrote:
Simone,
The issue you are seeing is very interesting - a message we add to a deque
disappear next time we check and according the log you provided there is no
code accessing the deque. It happens only for one specific message. All
the other messages work ok. Can you please gather logs so we could see what
is really happening with it?
And open a BZ fir tracking this too, please.
Thanks,
Piotr
26 kwi 2016 10:18 "Simone Tiraboschi" <stirabos(a)redhat.com> napisał(a):
> 2016-04-25 20:15 GMT+02:00 Nir Soffer <nsoffer(a)redhat.com>:
> > Hi all,
> >
> > We see random failures in the stomp tests for long time.
> >
> > I suggest we mark these tests as broken for now.
> >
> > See examples at the end of the message.
> >
> > Nir
> >
> > ----
> >
>
> > 17:14:24
> ======================================================================
> > 17:14:24 ERROR: test_event(False) (stompTests.StompTests)
> > 17:14:24
> ----------------------------------------------------------------------
> > 17:14:24 Traceback (most recent call last):
> > 17:14:24 File
> >
>
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testlib.py",
> > line 74, in wrapper
> > 17:14:24 return f(self, *args)
> > 17:14:24 File
> >
>
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/stompTests.py",
> > line 110, in test_event
> > 17:14:24 client.callMethod("event", [], str(uuid4()))
> > 17:14:24 File
> >
>
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/__init__.py",
> > line 340, in callMethod
> > 17:14:24 raise JsonRpcNoResponseError(methodName)
> > 17:14:24 JsonRpcNoResponseError: [-32605] No response for JSON-RPC
> > event request.
>
> I'd suggest to investigate this a bit more since it can hide a serious
> issue.
> I'm moving hosted-engine-setup from XMLRPC to JsonRPC and I'm facing
> exactly this kind of issue: it seams that some requests got lost and I
> just receive a JsonRpcNoResponseError after a long time.
> The real issue is that my request never reached VDSM getting lost
> somehow in the queuing mechanism.
>
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
--
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com
4:19 p.m.
On Fri, Apr 29, 2016 at 9:03 AM, Sandro Bonazzola <sbonazzo(a)redhat.com>
wrote:
On Tue, Apr 26, 2016 at 10:40 AM, Piotr Kliczewski <pkliczew(a)redhat.com>
wrote:
> Simone,
>
> The issue you are seeing is very interesting - a message we add to a
> deque disappear next time we check and according the log you provided there
> is no code accessing the deque. It happens only for one specific message.
> All the other messages work ok. Can you please gather logs so we could see
> what is really happening with it?
>
And open a BZ fir tracking this too, please.
We already have one [1]
[1] https://bugzilla.redhat.com/1321325
> Thanks,
> Piotr
> 26 kwi 2016 10:18 "Simone Tiraboschi" <stirabos(a)redhat.com>
napisał(a):
>
>> 2016-04-25 20:15 GMT+02:00 Nir Soffer <nsoffer(a)redhat.com>:
>> > Hi all,
>> >
>> > We see random failures in the stomp tests for long time.
>> >
>> > I suggest we mark these tests as broken for now.
>> >
>> > See examples at the end of the message.
>> >
>> > Nir
>> >
>> > ----
>> >
>>
>> > 17:14:24
>> ======================================================================
>> > 17:14:24 ERROR: test_event(False) (stompTests.StompTests)
>> > 17:14:24
>> ----------------------------------------------------------------------
>> > 17:14:24 Traceback (most recent call last):
>> > 17:14:24 File
>> >
>>
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testlib.py",
>> > line 74, in wrapper
>> > 17:14:24 return f(self, *args)
>> > 17:14:24 File
>> >
>>
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/stompTests.py",
>> > line 110, in test_event
>> > 17:14:24 client.callMethod("event", [], str(uuid4()))
>> > 17:14:24 File
>> >
>>
"/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/__init__.py",
>> > line 340, in callMethod
>> > 17:14:24 raise JsonRpcNoResponseError(methodName)
>> > 17:14:24 JsonRpcNoResponseError: [-32605] No response for JSON-RPC
>> > event request.
>>
>> I'd suggest to investigate this a bit more since it can hide a serious
>> issue.
>> I'm moving hosted-engine-setup from XMLRPC to JsonRPC and I'm facing
>> exactly this kind of issue: it seams that some requests got lost and I
>> just receive a JsonRpcNoResponseError after a long time.
>> The real issue is that my request never reached VDSM getting lost
>> somehow in the queuing mechanism.
>>
>
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
--
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com
2919
days inactive
2923
days old
4 comments
4 participants
participants (4)
-
Nir Soffer -
Piotr Kliczewski -
Sandro Bonazzola -
Simone Tiraboschi