[ovirt-devel] Re: [VDSM] VdsmClientTests.test_event_handler - flaky test?

On Tue, Mar 24, 2020 at 12:20 PM Martin Perina <mperina(a)redhat.com&gt; wrote: > > > > On Tue, Mar 24, 2020 at 11:11 AM Marcin Sobczyk <msobczyk(a)redhat.com&gt; wrote: >> >> >> >> On 3/23/20 4:32 PM, Nir Soffer wrote: >> > On Tue, Mar 17, 2020 at 4:26 PM Marcin Sobczyk <msobczyk(a)redhat.com&gt; wrote: >> >> Hi, >> >> >> >> sorry for the delay. I've posted [1] to disable all the mentioned tests >> >> in CI. >> > Thanks, I still see random failures in travis: >> Posted: https://gerrit.ovirt.org/#/c/107866/ >> >> > >> > =================================== FAILURES =================================== >> > _____________________ VdsmClientTests.test_missing_method ______________________ >> > self = <stomprpcclient_test.VdsmClientTests testMethod=test_missing_method> >> > @broken_on_ci("Fails randomly in CI") >> > >> > This disabled the tests only on oVirt CI (OVIRT_CI=1), but it does not >> > disable on >> > travis (TRAVIS_CI=1). >> > >> > Maybe we need to remove remove the default to avoid confusion. >> That would add quite a lot of boilerplate I guess. Also, given that we >> don't have >> gerrit-travis integration, it's jenkins that is the source of truth for >> us. Not really, some tests cannot run on jenkins or are flaky, but run fine on travis. For storage successful build on travis is required to get your patches merged. >> I would rather >> leave things as is. > > > +1 > > What exactly do we need travis for? As VDSM is primarily hosted on gerrit, we should be doing all CI stuff on jenkins. Current CI in jenkins does not work for us. Lot of storage tests are broken and skipped on jenkins but works fine on Travis. On Travis we have better environment, running each build in a new VM, and using containers for testing different distributions, so we control our test environment better. Look at typical run on jenkins: https://jenkins.ovirt.org/job/vdsm_standard-check-patch/19605/ Took 24 minutes. Expected failures: $ curl -s https://jenkins.ovirt.org/job/vdsm_standard-check-patch/19605//artifact/c... | grep XFAIL | wc -l 166 Same build on Travis: https://travis-ci.org/github/oVirt/vdsm/builds/666254744 Took 9 min 38 sec Expected failures: $ curl -s https://api.travis-ci.org/v3/job/666254745/log.txt | grep XFAIL | wc -l 22

You can see that travis builds are much faster. Also maintaining our travis build is much easier - basically single file, and some dockerfiles: $ find docker/ | wc -l 7 $ cat .travis.yml docker/* | wc -l 282 Compared with 57 files in automation/ and 1189 lines: $ find automation | wc -l 57 $ cat automation/* | wc -l 1189 > Travis integration makes sense only for github hosted projects, where integration with jenkins is less capable It makes sense for the reason above and others. For example qemu is using both Travis and other CIs enviromnets. Sanlock has also unofficial CI on Travis (and no other CI). We need Travis in gerrit, or move projects to github, or maybe gitlab. The recommended workflow for vsm and ovirt-imageio is to run the tests on Travis, and upload a patch to gerrit only when your Travis build is working. This can save a lot of time and avoids unwanted notifications from jenkins to reviewers when your build fails. >> > def test_missing_method(self): >> > with self._create_client() as client: >> > with self.assertRaises(ServerError) as ex: >> >> client.Test.missingMethod() >> > lib/yajsonrpc/stomprpcclient_test.py:159: >> > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >> > def _call(self, namespace, method_name, **kwargs): >> > """ >> > Client call method, executes a given command >> > >> > Args: >> > namespace (string): namespace name >> > method_name (string): method name >> > **kwargs: Arbitrary keyword arguments >> > >> > Returns: >> > method result >> > >> > Raises: >> > ClientError: in case of an error in the protocol. >> > TimeoutError: if there is no response after a pre configured time. >> > ServerError: in case of an error while executing the command >> > """ >> > method = namespace + "." + method_name >> > timeout = kwargs.pop("_timeout", self._default_timeout) >> > >> > req = yajsonrpc.JsonRpcRequest( >> > method, kwargs, reqId=str(uuid.uuid4())) >> > >> > try: >> > responses = self._client.call( >> > req, timeout=timeout, flow_id=self._flow_id) >> > except EnvironmentError as e: >> > raise ClientError(method, kwargs, e) >> > >> > if not responses: >> >> raise TimeoutError(method, kwargs, timeout) >> > E vdsm.client.TimeoutError: Request Test.missingMethod with >> > args {} timed out after 3 seconds >> > ../lib/vdsm/client.py:294: TimeoutError >> > ------------------------------ Captured log call ------------------------------- >> > ERROR vds.dispatcher:betterAsyncore.py:179 uncaptured python >> > exception, closing channel <yajsonrpc.betterAsyncore.Dispatcher >> > 127.0.0.1:60800 at 0x7f07c2297438> (<class >> > 'ValueError'>:'b'ept-version:1.2'' contains illegal character ':' >> > [/usr/lib64/python3.6/asyncore.py|readwrite|108] >> > [/usr/lib64/python3.6/asyncore.py|handle_read_event|423] >> > [/vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71] >> > [/vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168] >> > [/vdsm/lib/vdsm/protocoldetector.py|handle_read|129] >> > [/vdsm/lib/yajsonrpc/stompserver.py|handle_socket|413] >> > [/vdsm/lib/vdsm/rpc/bindingjsonrpc.py|add_socket|54] >> > [/vdsm/lib/yajsonrpc/stompserver.py|createListener|379] >> > [/vdsm/lib/yajsonrpc/stompserver.py|StompListener|345] >> > [/vdsm/lib/yajsonrpc/betterAsyncore.py|__init__|47] >> > [/vdsm/lib/yajsonrpc/betterAsyncore.py|switch_implementation|86] >> > [/vdsm/lib/yajsonrpc/stompserver.py|init|363] >> > [/vdsm/lib/vdsm/rpc/bindingjsonrpc.py|_onAccept|57] >> > [/vdsm/lib/yajsonrpc/stomp.py|set_message_handler|645] >> > [/usr/lib64/python3.6/asyncore.py|handle_read_event|423] >> > [/vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71] >> > [/vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168] >> > [/vdsm/lib/yajsonrpc/stomp.py|handle_read|421] >> > [/vdsm/lib/yajsonrpc/stomp.py|parse|323] >> > [/vdsm/lib/yajsonrpc/stomp.py|_parse_command|245] >> > [/vdsm/lib/yajsonrpc/stomp.py|decode_value|167]) >> > WARNING vds.dispatcher:betterAsyncore.py:179 unhandled close event >> > >> >> Marcin >> >> >> >> [1] https://gerrit.ovirt.org/#/c/107711/ >> >> >> >> On 3/17/20 1:15 PM, Nir Soffer wrote: >> >>> Another failure today. >> >>> >> >>> Marcin, it is time to either fix or disable these flaky tests. >> >>> >> >>> _____________________ VdsmClientTests.test_bad_parameters ______________________ >> >>> >> >>> self = <stomprpcclient_test.VdsmClientTests testMethod=test_bad_parameters> >> >>> >> >>> def test_bad_parameters(self): >> >>> with self._create_client() as client: >> >>> with self.assertRaises(ServerError) as ex: >> >>>> client.Test.echo() >> >>> lib/yajsonrpc/stomprpcclient_test.py:173: >> >>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >> >>> >> >>> def _call(self, namespace, method_name, **kwargs): >> >>> """ >> >>> Client call method, executes a given command >> >>> >> >>> Args: >> >>> namespace (string): namespace name >> >>> method_name (string): method name >> >>> **kwargs: Arbitrary keyword arguments >> >>> >> >>> Returns: >> >>> method result >> >>> >> >>> Raises: >> >>> ClientError: in case of an error in the protocol. >> >>> TimeoutError: if there is no response after a pre configured time. >> >>> ServerError: in case of an error while executing the command >> >>> """ >> >>> method = namespace + "." + method_name >> >>> timeout = kwargs.pop("_timeout", self._default_timeout) >> >>> >> >>> req = yajsonrpc.JsonRpcRequest( >> >>> method, kwargs, reqId=str(uuid.uuid4())) >> >>> >> >>> try: >> >>> responses = self._client.call( >> >>> req, timeout=timeout, flow_id=self._flow_id) >> >>> except EnvironmentError as e: >> >>> raise ClientError(method, kwargs, e) >> >>> >> >>> if not responses: >> >>>> raise TimeoutError(method, kwargs, timeout) >> >>> E vdsm.client.TimeoutError: Request Test.echo with args {} >> >>> timed out after 3 seconds >> >>> >> >>> ../lib/vdsm/client.py:294: TimeoutError >> >>> ------------------------------ Captured log call ------------------------------- >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>> ERROR vds.dispatcher:betterAsyncore.py:179 uncaptured python >> >>> exception, closing channel <yajsonrpc.betterAsyncore.Dispatcher >> >>> ('::1', 46108, 0, 0) at 0x7f2c3201e438> (<class >> >>> 'ValueError'>:'b'ept-version:1.2'' contains illegal character ':' >> >>> [/usr/lib64/python3.6/asyncore.py|readwrite|108] >> >>> [/usr/lib64/python3.6/asyncore.py|handle_read_event|423] >> >>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71] >> >>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168] >> >>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/protocoldetector.py|handle_read|129] >> >>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|handle_socket|413] >> >>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/rpc/bindingjsonrpc.py|add_socket|54] >> >>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|createListener|379] >> >>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|StompListener|345] >> >>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|__init__|47] >> >>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|switch_implementation|86] >> >>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|init|363] >> >>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/rpc/bindingjsonrpc.py|_onAccept|57] >> >>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|set_message_handler|645] >> >>> [/usr/lib64/python3.6/asyncore.py|handle_read_event|423] >> >>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71] >> >>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168] >> >>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|handle_read|421] >> >>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|parse|323] >> >>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|_parse_command|245] >> >>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|decode_value|167]) >> >>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled close event >> >>> ERROR root:concurrent.py:267 FINISH thread <Thread(JsonRpc >> >>> (StompReactor), started daemon 139827525695232)> failed >> >>> Traceback (most recent call last): >> >>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/common/concurrent.py", >> >>> line 260, in run >> >>> ret = func(*args, **kwargs) >> >>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py", >> >>> line 393, in process_requests >> >>> self._reactor.process_requests() >> >>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py", >> >>> line 238, in process_requests >> >>> timeout=self._get_timeout(self._map), >> >>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py", >> >>> line 253, in _get_timeout >> >>> interval = disp.next_check_interval() >> >>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py", >> >>> line 99, in next_check_interval >> >>> return getattr(self.__impl, "next_check_interval", default_func)() >> >>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py", >> >>> line 486, in next_check_interval >> >>> self.handle_timeout() >> >>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py", >> >>> line 451, in handle_timeout >> >>> self._frame_handler.handle_timeout(self) >> >>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompclient.py", >> >>> line 130, in handle_timeout >> >>> dispatcher._on_timeout) >> >>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py", >> >>> line 630, in reconnect >> >>> AsyncDispatcher(self, self._async_client, count=count)) >> >>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py", >> >>> line 271, in reconnect >> >>> dispatcher.create_socket(address, sslctx) >> >>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py", >> >>> line 112, in create_socket >> >>> sock = sslctx.wrapSocket(sock) >> >>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/sslutils.py", >> >>> line 113, in wrapSocket >> >>> ca_certs=self.ca_certs) >> >>> File "/usr/lib64/python3.6/ssl.py", line 1114, in wrap_socket >> >>> ciphers=ciphers) >> >>> File "/usr/lib64/python3.6/ssl.py", line 704, in __init__ >> >>> self._context.load_verify_locations(ca_certs) >> >>> FileNotFoundError: [Errno 2] No such file or directory >> >>> >> >>> On Thu, Mar 12, 2020 at 5:34 PM Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >> >>>> I had 2 of these unrelated failures today. Would be nice to mark this >> >>>> test as broken on CI. >> >>>> >> >>>> https://jenkins.ovirt.org/job/vdsm_standard-check-patch/19353//artifact/c... >> >>>> https://jenkins.ovirt.org/job/vdsm_standard-check-patch/19354//artifact/c... >> >>>> >> >>>> =================================== FAILURES =================================== >> >>>> ______________________ VdsmClientTests.test_event_handler ______________________ >> >>>> >> >>>> self = <stomprpcclient_test.VdsmClientTests testMethod=test_event_handler> >> >>>> >> >>>> def test_event_handler(self): >> >>>> with self._create_client() as client: >> >>>> event_queue = queue.Queue() >> >>>> >> >>>> sub_id = client.subscribe(EVENT_TOPIC, event_queue) >> >>>>> client.Test.sendEvent() >> >>>> lib/yajsonrpc/stomprpcclient_test.py:215: >> >>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >> >>>> >> >>>> self = <stomprpcclient_test._MockedClient object at 0x7fe19f6db1d0> >> >>>> namespace = 'Test', method_name = 'sendEvent', kwargs = {} >> >>>> method = 'Test.sendEvent', timeout = 3 >> >>>> req = {'jsonrpc': '2.0', 'method': 'Test.sendEvent', 'params': {}, >> >>>> 'id': 'c3985771-6f77-4431-9ed6-15082063a2d5'} >> >>>> responses = None >> >>>> >> >>>> def _call(self, namespace, method_name, **kwargs): >> >>>> """ >> >>>> Client call method, executes a given command >> >>>> >> >>>> Args: >> >>>> namespace (string): namespace name >> >>>> method_name (string): method name >> >>>> **kwargs: Arbitrary keyword arguments >> >>>> >> >>>> Returns: >> >>>> method result >> >>>> >> >>>> Raises: >> >>>> ClientError: in case of an error in the protocol. >> >>>> TimeoutError: if there is no response after a pre configured time. >> >>>> ServerError: in case of an error while executing the command >> >>>> """ >> >>>> method = namespace + "." + method_name >> >>>> timeout = kwargs.pop("_timeout", self._default_timeout) >> >>>> >> >>>> req = yajsonrpc.JsonRpcRequest( >> >>>> method, kwargs, reqId=str(uuid.uuid4())) >> >>>> >> >>>> try: >> >>>> responses = self._client.call( >> >>>> req, timeout=timeout, flow_id=self._flow_id) >> >>>> except EnvironmentError as e: >> >>>> raise ClientError(method, kwargs, e) >> >>>> >> >>>> if not responses: >> >>>>> raise TimeoutError(method, kwargs, timeout) >> >>>> E vdsm.client.TimeoutError: Request Test.sendEvent with args >> >>>> {} timed out after 3 seconds >> >>>> >> >>>> ../lib/vdsm/client.py:294: TimeoutError >> >>>> ------------------------------ Captured log call ------------------------------- >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event >> >>>> ERROR vds.dispatcher:betterAsyncore.py:179 uncaptured python >> >>>> exception, closing channel <yajsonrpc.betterAsyncore.Dispatcher >> >>>> ('::1', 36398, 0, 0) at 0x7fe19f6db208> (<class >> >>>> 'ValueError'>:'b'ept-version:1.2'' contains illegal character ':' >> >>>> [/usr/lib64/python3.6/asyncore.py|readwrite|108] >> >>>> [/usr/lib64/python3.6/asyncore.py|handle_read_event|423] >> >>>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71] >> >>>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168] >> >>>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/protocoldetector.py|handle_read|129] >> >>>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|handle_socket|413] >> >>>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/rpc/bindingjsonrpc.py|add_socket|54] >> >>>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|createListener|379] >> >>>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|StompListener|345] >> >>>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|__init__|47] >> >>>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|switch_implementation|86] >> >>>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|init|363] >> >>>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/rpc/bindingjsonrpc.py|_onAccept|57] >> >>>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|set_message_handler|645] >> >>>> [/usr/lib64/python3.6/asyncore.py|handle_read_event|423] >> >>>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71] >> >>>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168] >> >>>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|handle_read|421] >> >>>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|parse|323] >> >>>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|_parse_command|245] >> >>>> [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|decode_value|167]) >> >>>> WARNING vds.dispatcher:betterAsyncore.py:179 unhandled close event >> >>>> ERROR root:concurrent.py:267 FINISH thread <Thread(JsonRpc >> >>>> (StompReactor), started daemon 140606575712000)> failed >> >>>> Traceback (most recent call last): >> >>>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/common/concurrent.py", >> >>>> line 260, in run >> >>>> ret = func(*args, **kwargs) >> >>>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py", >> >>>> line 393, in process_requests >> >>>> self._reactor.process_requests() >> >>>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py", >> >>>> line 238, in process_requests >> >>>> timeout=self._get_timeout(self._map), >> >>>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py", >> >>>> line 253, in _get_timeout >> >>>> interval = disp.next_check_interval() >> >>>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py", >> >>>> line 99, in next_check_interval >> >>>> return getattr(self.__impl, "next_check_interval", default_func)() >> >>>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py", >> >>>> line 486, in next_check_interval >> >>>> self.handle_timeout() >> >>>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py", >> >>>> line 451, in handle_timeout >> >>>> self._frame_handler.handle_timeout(self) >> >>>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompclient.py", >> >>>> line 130, in handle_timeout >> >>>> dispatcher._on_timeout) >> >>>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py", >> >>>> line 630, in reconnect >> >>>> AsyncDispatcher(self, self._async_client, count=count)) >> >>>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py", >> >>>> line 271, in reconnect >> >>>> dispatcher.create_socket(address, sslctx) >> >>>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py", >> >>>> line 112, in create_socket >> >>>> sock = sslctx.wrapSocket(sock) >> >>>> File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/sslutils.py", >> >>>> line 113, in wrapSocket >> >>>> ca_certs=self.ca_certs) >> >>>> File "/usr/lib64/python3.6/ssl.py", line 1114, in wrap_socket >> >>>> ciphers=ciphers) >> >>>> File "/usr/lib64/python3.6/ssl.py", line 704, in __init__ >> >>>> self._context.load_verify_locations(ca_certs) >> >>>> FileNotFoundError: [Errno 2] No such file or directory >> >>>> =============================== warnings summary =============================== >> >>>> /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/lib/lib/python3.6/site-packages/_pytest/mark/structures.py:334 >> >>>> /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/lib/lib/python3.6/site-packages/_pytest/mark/structures.py:334: >> >>>> PytestUnknownMarkWarning: Unknown pytest.mark.stress - is this a typo? >> >>>> You can register custom marks to avoid this warning - for details, >> >>>> see https://docs.pytest.org/en/latest/mark.html >> >>>> PytestUnknownMarkWarning, >> >>>> >> >>>> /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/lib/lib/python3.6/site-packages/_pytest/mark/structures.py:334 >> >>>> /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/lib/lib/python3.6/site-packages/_pytest/mark/structures.py:334: >> >>>> PytestUnknownMarkWarning: Unknown pytest.mark.slow - is this a typo? >> >>>> You can register custom marks to avoid this warning - for details, see >> >>>> https://docs.pytest.org/en/latest/mark.html >> >>>> PytestUnknownMarkWarning, >> >>>> >> >>>> /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/v2v.py:79 >> >>>> /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/v2v.py:79: >> >>>> DeprecationWarning: invalid escape sequence \d >> >>>> _SSH_AUTH_RE = b'(SSH_AUTH_SOCK)=([^;]+).*;\nSSH_AGENT_PID=(\d+)' >> >>>> >> >>>> /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/v2v.py:1421 >> >>>> /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/v2v.py:1421: >> >>>> DeprecationWarning: invalid escape sequence \^ >> >>>> '(?P<m2_base>[0-9]+){sp}\^{sp}(?P<m2_exp>{exp}))'.format( >> >>>> >> >>>> /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/lib/lib/python3.6/site-packages/_pytest/mark/structures.py:334 >> >>>> /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/lib/lib/python3.6/site-packages/_pytest/mark/structures.py:334: >> >>>> PytestUnknownMarkWarning: Unknown pytest.mark.xpass - is this a typo? >> >>>> You can register custom marks to avoid this warning - for details, see >> >>>> https://docs.pytest.org/en/latest/mark.html >> >>>> PytestUnknownMarkWarning, >> >>>> >> >>>> tests/lib/osinfo_test.py::test_package_versions >> >>>> /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/osinfo.py:284: >> >>>> UnicodeWarning: decode() called on unicode string, see >> >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1693751 >> >>>> 'version': mi['version'].decode('utf-8'), >> >>>> >> >>>> tests/lib/osinfo_test.py::test_package_versions >> >>>> /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/osinfo.py:285: >> >>>> UnicodeWarning: decode() called on unicode string, see >> >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1693751 >> >>>> 'release': mi['release'].decode('utf-8'), >> >>>> >> >>>> -- Docs: https://docs.pytest.org/en/latest/warnings.html >> >>>> >> >>>> ----------- coverage: platform linux, python 3.6.8-final-0 ----------- >> >>>> Coverage HTML written to dir htmlcov-lib >> >>>> >> >>>> ========================== slowest 10 test durations =========================== >> >>>> 18.29s call >> >>>> tests/lib/yajsonrpc/stomprpcclient_test.py::VdsmClientTests::test_event_handler >> >>>> 2.96s call tests/lib/protocoldetector_test.py::AcceptorTests::test_reject_very_slow_client_concurrency(True) >> >>>> 2.90s call tests/lib/protocoldetector_test.py::AcceptorTests::test_reject_very_slow_client_concurrency(False) >> >>>> 2.75s call tests/lib/protocoldetector_test.py::AcceptorTests::test_reject_very_slow_client(False) >> >>>> 2.63s call tests/lib/protocoldetector_test.py::AcceptorTests::test_reject_very_slow_client(True) >> >>>> 1.90s call tests/pywatch_test.py::TestPyWatch::test_timeout_backtrace >> >>>> 1.44s call tests/pywatch_test.py::TestPyWatch::test_timeout_output >> >>>> 1.19s call tests/pywatch_test.py::TestPyWatch::test_kill_grandkids >> >>>> 0.93s call tests/lib/protocoldetector_test.py::AcceptorTests::test_detect_slow_client(True) >> >>>> 0.90s call tests/lib/yajsonrpc/stomp_test.py::StompTests::test_echo(16384, >> >>>> False) >> >>>> =========================== short test summary info ============================ >> _______________________________________________ >> Devel mailing list -- devel(a)ovirt.org >> To unsubscribe send an email to devel-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/Q5BUNIRQ7AE... > > > > -- > Martin Perina > Manager, Software Engineering > Red Hat Czech s.r.o.