9:45 p.m.
Marcin Sobczyk <msobczyk(a)redhat.com> writes:
Hi,
On 12/7/19 7:52 AM, Martin Perina wrote:
> Marcin, could you please investigate?
>
These failures are the already known race/timeout issues with our
yajsonrpc/stomp tests.
Since we switched to dynamic SSL key-cert generation in [2] they just
manifest in a different way.
I posted [1] to disable these tests for now.
Thanks, Marcin, looks like a good step now to avoid the failures.
[1] https://gerrit.ovirt.org/105431
[2]
https://gerrit.ovirt.org/#/q/status:merged+project:vdsm+branch:master+top...
>
> On Fri, 6 Dec 2019, 19:50 Milan Zamazal, <mzamazal(a)redhat.com
> <mailto:mzamazal@redhat.com>> wrote:
>
> [Sorry, sent an unfinished mail by mistake.]
>
> Milan Zamazal <mzamazal(a)redhat.com <mailto:mzamazal@redhat.com>>
> writes:
>
> > Hi, this seems to be a frequent error in Jenkins run recently:
> >
> >
> > =================================== FAILURES
> ===================================
> > ______________________ VdsmClientTests.test_failing_call
> _______________________
> >
> > self = <stomprpcclient_test.VdsmClientTests
> testMethod=test_failing_call>
> >
> > def test_failing_call(self):
> > with self._create_client() as client:
> > with self.assertRaises(ServerError) as ex:
> >> client.Test.failingCall()
> >
> > lib/yajsonrpc/stomprpcclient_test.py:144:
> > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _
> >
> > def _call(self, namespace, method_name, **kwargs):
> > """
> > Client call method, executes a given command
> >
> > Args:
> > namespace (string): namespace name
> > method_name (string): method name
> > **kwargs: Arbitrary keyword arguments
> >
> > Returns:
> > method result
> >
> > Raises:
> > ClientError: in case of an error in the protocol.
> > TimeoutError: if there is no response after a pre
> configured time.
> > ServerError: in case of an error while executing the
> command
> > """
> > method = namespace + "." + method_name
> > timeout = kwargs.pop("_timeout", self._default_timeout)
> >
> > req = yajsonrpc.JsonRpcRequest(
> > method, kwargs, reqId=str(uuid.uuid4()))
> >
> > try:
> > responses = self._client.call(
> > req, timeout=timeout, flow_id=self._flow_id)
> > except EnvironmentError as e:
> > raise ClientError(method, kwargs, e)
> >
> > if not responses:
> >> raise TimeoutError(method, kwargs, timeout)
> > E vdsm.client.TimeoutError: Request Test.failingCall
> with args {} timed out after 3 seconds
> >
> > ../lib/vdsm/client.py:294: TimeoutError
> > ------------------------------ Captured log call
> -------------------------------
> > ERROR vds.dispatcher:betterAsyncore.py:179 uncaptured python
> exception, closing channel <yajsonrpc.betterAsyncore.Dispatcher
> ('::1', 47428, 0, 0) at 0x7f48ddc47d10> (<class
> 'ValueError'>:'b'ept-version:1.2'' contains illegal
character ':'
> [/usr/lib64/python3.7/asyncore.py|readwrite|108]
> [/usr/lib64/python3.7/asyncore.py|handle_read_event|422]
>
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71]
>
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168]
>
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/protocoldetector.py|handle_read|129]
>
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|handle_socket|413]
>
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/rpc/bindingjsonrpc.py|add_socket|54]
>
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|createListener|379]
>
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|StompListener|345]
>
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|__init__|47]
>
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|switch_implementation|86]
>
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|init|363]
>
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/rpc/bindingjsonrpc.py|_onAccept|57]
>
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|set_message_handler|645]
> [/usr/lib64/python3.7/asyncore.py|handle_read_event|422]
>
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71]
>
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168]
>
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|handle_read|421]
>
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|parse|323]
>
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|_parse_command|245]
>
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|decode_value|167])
> > WARNING vds.dispatcher:betterAsyncore.py:179 unhandled close event
> > ERROR root:concurrent.py:267 FINISH thread <Thread(JsonRpc
> (StompReactor), started daemon 139950158767872)> failed
> > Traceback (most recent call last):
> > File
>
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/common/concurrent.py",
> line 260, in run
> > ret = func(*args, **kwargs)
> > File
>
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py",
> line 393, in process_requests
> > self._reactor.process_requests()
> > File
>
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py",
> line 238, in process_requests
> > timeout=self._get_timeout(self._map),
> > File
>
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py",
> line 253, in _get_timeout
> > interval = disp.next_check_interval()
> > File
>
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py",
> line 99, in next_check_interval
> > return getattr(self.__impl, "next_check_interval",
> default_func)()
> > File
>
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py",
> line 486, in next_check_interval
> > self.handle_timeout()
> > File
>
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py",
> line 451, in handle_timeout
> > self._frame_handler.handle_timeout(self)
> > File
>
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompclient.py",
> line 130, in handle_timeout
> > dispatcher._on_timeout)
> > File
>
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py",
> line 630, in reconnect
> > AsyncDispatcher(self, self._async_client, count=count))
> > File
>
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py",
> line 271, in reconnect
> > dispatcher.create_socket(address, sslctx)
> > File
>
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py",
> line 112, in create_socket
> > sock = sslctx.wrapSocket(sock)
> > File
>
"/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/sslutils.py",
> line 113, in wrapSocket
> > ca_certs=self.ca_certs)
> > File "/usr/lib64/python3.7/ssl.py", line 1230, in wrap_socket
> > context.load_verify_locations(ca_certs)
> > FileNotFoundError: [Errno 2] No such file or directory
> > _____________________ VdsmClientTests.test_missing_method
> ______________________
> >
> > self = <stomprpcclient_test.VdsmClientTests
> testMethod=test_missing_method>
> >
> > def test_missing_method(self):
> > with self._create_client() as client:
> > with self.assertRaises(ServerError) as ex:
> >> client.Test.missingMethod()
> >
> > lib/yajsonrpc/stomprpcclient_test.py:155:
> > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _
> >
> > def _call(self, namespace, method_name, **kwargs):
> > """
> > Client call method, executes a given command
> >
> > Args:
> > namespace (string): namespace name
> > method_name (string): method name
> > **kwargs: Arbitrary keyword arguments
> >
> > Returns:
> > method result
> >
> > Raises:
> > ClientError: in case of an error in the protocol.
> > TimeoutError: if there is no response after a pre
> configured time.
> > ServerError: in case of an error while executing the
> command
> > """
> > method = namespace + "." + method_name
> > timeout = kwargs.pop("_timeout", self._default_timeout)
> >
> > req = yajsonrpc.JsonRpcRequest(
> > method, kwargs, reqId=str(uuid.uuid4()))
> >
> > try:
> > responses = self._client.call(
> > req, timeout=timeout, flow_id=self._flow_id)
> > except EnvironmentError as e:
> > raise ClientError(method, kwargs, e)
> >
> > if not responses:
> >> raise TimeoutError(method, kwargs, timeout)
> > E vdsm.client.TimeoutError: Request Test.missingMethod
> with args {} timed out after 3 seconds
> >
> > ../lib/vdsm/client.py:294: TimeoutError
>
> See e.g.
>
https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-...
>
> If we can't fix the CI, can we disable the test temporarily?
> I experience very frequent Vdsm CI failures these days.
>
> Thanks,
> Milan
> _______________________________________________
> Devel mailing list -- devel(a)ovirt.org <mailto:devel@ovirt.org>
> To unsubscribe send an email to devel-leave(a)ovirt.org
> <mailto:devel-leave@ovirt.org>
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/BOM7KFS5BFS...
>