On Wed, Jun 27, 2018 at 5:01 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > This tests used to fail in the past, but since we fixed it or the code > > it never failed. > > > Maybe the slave was overloaded? > > Very possible. Can you paste a link to the job which failed?

> > *14:19:02* ======================================================================*14:19:02* ERROR: > > *14:19:02* ----------------------------------------------------------------------*14:19:02* Traceback (most recent call last):*14:19:02* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/testlib.py", line 143, in wrapper*14:19:02* return f(self, *args)*14:19:02* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/stomp_test.py", line 95, in test_echo*14:19:02* str(uuid4())),*14:19:02* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/lib/yajsonrpc/jsonrpcclient.py", line 77, in callMethod*14:19:02* raise exception.JsonRpcNoResponseError(method=methodName)*14:19:02* JsonRpcNoResponseError: No response for JSON-RPC request: {'method': 'echo'}*14:19:02* -------------------- >> begin captured logging << --------------------*14:19:02* 2018-06-27 14:17:54,524 DEBUG (MainThread) [vds.MultiProtocolAcceptor] Creating socket (host='::1', port=0, family=10, socketype=1, proto=6) (protocoldetector:225)*14:19:02* 2018-06-27 14:17:54,526 INFO (MainThread) [vds.MultiProtocolAcceptor] Listening at ::1:36713 (protocoldetector:183)*14:19:02* 2018-06-27 14:17:54,535 DEBUG (MainThread) [Scheduler] Starting scheduler test.Scheduler (schedule:98)*14:19:02* 2018-06-27 14:17:54,537 DEBUG (test.Scheduler) [Scheduler] START thread <Thread(test.Scheduler, started daemon 140562082535168)> (func=<bound method Scheduler._run of <vdsm.schedule.Scheduler object at 0x7fd74ca00390>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,538 DEBUG (test.Scheduler) [Scheduler] started (schedule:140)*14:19:02* 2018-06-27 14:17:54,546 DEBUG (JsonRpc (StompReactor)) [root] START thread <Thread(JsonRpc (StompReactor), started daemon 140562629256960)> (func=<bound method StompReactor.process_requests of <yajsonrpc.stompserver.StompReactor object at 0x7fd74ca006d0>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,547 DEBUG (MainThread) [Executor] Starting executor (executor:128)*14:19:02* 2018-06-27 14:17:54,549 DEBUG (MainThread) [Executor] Starting worker jsonrpc/0 (executor:286)*14:19:02* 2018-06-27 14:17:54,553 DEBUG (jsonrpc/0) [Executor] START thread <Thread(jsonrpc/0, started daemon 140562612471552)> (func=<bound method _Worker._run of <Worker name=jsonrpc/0 waiting task#=0 at 0x7fd74ca00650>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,554 DEBUG (jsonrpc/0) [Executor] Worker started (executor:298)*14:19:02* 2018-06-27 14:17:54,557 DEBUG (MainThread) [Executor] Starting worker jsonrpc/1 (executor:286)*14:19:02* 2018-06-27 14:17:54,558 DEBUG (MainThread) [Executor] Starting worker jsonrpc/2 (executor:286)*14:19:02* 2018-06-27 14:17:54,559 DEBUG (jsonrpc/2) [Executor] START thread <Thread(jsonrpc/2, started daemon 140562620864256)> (func=<bound method _Worker._run of <Worker name=jsonrpc/2 waiting task#=0 at 0x7fd74c9fb350>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,560 DEBUG (jsonrpc/2) [Executor] Worker started (executor:298)*14:19:02* 2018-06-27 14:17:54,561 DEBUG (MainThread) [Executor] Starting worker jsonrpc/3 (executor:286)*14:19:02* 2018-06-27 14:17:54,562 DEBUG (jsonrpc/3) [Executor] START thread <Thread(jsonrpc/3, started daemon 140562124498688)> (func=<bound method _Worker._run of <Worker name=jsonrpc/3 waiting task#=0 at 0x7fd74bc80290>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,563 DEBUG (jsonrpc/3) [Executor] Worker started (executor:298)*14:19:02* 2018-06-27 14:17:54,564 DEBUG (MainThread) [Executor] Starting worker jsonrpc/4 (executor:286)*14:19:02* 2018-06-27 14:17:54,565 DEBUG (jsonrpc/4) [Executor] START thread <Thread(jsonrpc/4, started daemon 140562116105984)> (func=<bound method _Worker._run of <Worker name=jsonrpc/4 waiting task#=0 at 0x7fd74bc80790>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,566 DEBUG (jsonrpc/4) [Executor] Worker started (executor:298)*14:19:02* 2018-06-27 14:17:54,568 DEBUG (jsonrpc/1) [Executor] START thread <Thread(jsonrpc/1, started daemon 140562132891392)> (func=<bound method _Worker._run of <Worker name=jsonrpc/1 waiting task#=0 at 0x7fd74c9fb690>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,569 DEBUG (jsonrpc/1) [Executor] Worker started (executor:298)*14:19:02* 2018-06-27 14:17:54,569 DEBUG (MainThread) [Executor] Starting worker jsonrpc/5 (executor:286)*14:19:02* 2018-06-27 14:17:54,570 DEBUG (jsonrpc/5) [Executor] START thread <Thread(jsonrpc/5, started daemon 140562107713280)> (func=<bound method _Worker._run of <Worker name=jsonrpc/5 waiting task#=0 at 0x7fd74bc80090>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,571 DEBUG (jsonrpc/5) [Executor] Worker started (executor:298)*14:19:02* 2018-06-27 14:17:54,572 DEBUG (MainThread) [Executor] Starting worker jsonrpc/6 (executor:286)*14:19:02* 2018-06-27 14:17:54,580 DEBUG (MainThread) [Executor] Starting worker jsonrpc/7 (executor:286)*14:19:02* 2018-06-27 14:17:54,603 DEBUG (jsonrpc/6) [Executor] START thread <Thread(jsonrpc/6, started daemon 140562099320576)> (func=<bound method _Worker._run of <Worker name=jsonrpc/6 waiting task#=0 at 0x7fd74d0dacd0>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,604 DEBUG (jsonrpc/6) [Executor] Worker started (executor:298)*14:19:02* 2018-06-27 14:17:54,615 DEBUG (jsonrpc/7) [Executor] START thread <Thread(jsonrpc/7, started daemon 140562090927872)> (func=<bound method _Worker._run of <Worker name=jsonrpc/7 waiting task#=0 at 0x7fd74bfe0310>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,616 DEBUG (JsonRpcServer) [root] START thread <Thread(JsonRpcServer, started daemon 140561730238208)> (func=<bound method JsonRpcServer.serve_requests of <yajsonrpc.JsonRpcServer object at 0x7fd74ca00690>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,618 DEBUG (MainThread) [vds.MultiProtocolAcceptor] Adding detector <yajsonrpc.stompserver.StompDetector instance at 0x7fd74c28cc68> (protocoldetector:210)*14:19:02* 2018-06-27 14:17:54,625 INFO (Detector thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:52432 (protocoldetector:61)*14:19:02* 2018-06-27 14:17:54,626 DEBUG (Detector thread) [ProtocolDetector.Detector] Using required_size=11 (protocoldetector:89)*14:19:02* 2018-06-27 14:17:54,627 DEBUG (jsonrpc/7) [Executor] Worker started (executor:298)*14:19:02* 2018-06-27 14:17:54,634 DEBUG (MainThread) [jsonrpc.AsyncoreClient] Sending response (stompclient:293)*14:19:02* 2018-06-27 14:17:54,634 DEBUG (Client ::1:36713) [root] START thread <Thread(Client ::1:36713, started daemon 140561713452800)> (func=<bound method Reactor.process_requests of <yajsonrpc.betterAsyncore.Reactor object at 0x7fd74bdb1d90>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,641 INFO (Detector thread) [ProtocolDetector.Detector] Detected protocol stomp from ::1:52432 (protocoldetector:125)*14:19:02* 2018-06-27 14:17:54,645 INFO (Detector thread) [Broker.StompAdapter] Processing CONNECT request (stompserver:94)*14:19:02* 2018-06-27 14:17:54,648 DEBUG (Client ::1:36713) [yajsonrpc.protocols.stomp.AsyncClient] Stomp connection established (stompclient:137)*14:19:02* 2018-06-27 14:17:54,651 DEBUG (jsonrpc/0) [jsonrpc.JsonRpcServer] Calling 'echo' in bridge with [u'Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut en'] (__init__:328)*14:19:02* 2018-06-27 14:17:54,652 DEBUG (jsonrpc/0) [jsonrpc.JsonRpcServer] Return 'echo' in bridge with Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut en (__init__:355)*14:19:02* 2018-06-27 14:17:54,653 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call echo succeeded in 0.01 seconds (__init__:311)*14:19:02* 2018-06-27 14:17:54,661 INFO (Detector thread) [Broker.StompAdapter] Subscribe command received (stompserver:123)*14:19:02* 2018-06-27 14:17:54,662 DEBUG (Detector thread) [protocoldetector.StompDetector] Stomp detected from ('::1', 52432) (stompserver:412)*14:19:02* 2018-06-27 14:18:09,649 DEBUG (MainThread) [vds.MultiProtocolAcceptor] Stopping Acceptor (protocoldetector:214)*14:19:02* 2018-06-27 14:18:09,650 INFO (MainThread) [jsonrpc.JsonRpcServer] Stopping JsonRPC Server (__init__:441)*14:19:02* 2018-06-27 14:18:09,651 DEBUG (MainThread) [Executor] Stopping executor (executor:137)*14:19:02* 2018-06-27 14:18:09,652 DEBUG (Client ::1:36713) [root] FINISH thread <Thread(Client ::1:36713, started daemon 140561713452800)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,652 DEBUG (JsonRpcServer) [root] FINISH thread <Thread(JsonRpcServer, started daemon 140561730238208)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,654 DEBUG (JsonRpc (StompReactor)) [root] FINISH thread <Thread(JsonRpc (StompReactor), started daemon 140562629256960)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,655 DEBUG (jsonrpc/2) [Executor] Worker stopped (executor:303)*14:19:02* 2018-06-27 14:18:09,658 DEBUG (jsonrpc/3) [Executor] Worker stopped (executor:303)*14:19:02* 2018-06-27 14:18:09,659 DEBUG (jsonrpc/4) [Executor] Worker stopped (executor:303)*14:19:02* 2018-06-27 14:18:09,660 DEBUG (jsonrpc/1) [Executor] Worker stopped (executor:303)*14:19:02* 2018-06-27 14:18:09,660 DEBUG (jsonrpc/5) [Executor] Worker stopped (executor:303)*14:19:02* 2018-06-27 14:18:09,660 DEBUG (jsonrpc/6) [Executor] Worker stopped (executor:303)*14:19:02* 2018-06-27 14:18:09,665 DEBUG (jsonrpc/6) [Executor] FINISH thread <Thread(jsonrpc/6, started daemon 140562099320576)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,661 DEBUG (jsonrpc/7) [Executor] Worker stopped (executor:303)*14:19:02* 2018-06-27 14:18:09,662 DEBUG (jsonrpc/2) [Executor] FINISH thread <Thread(jsonrpc/2, started daemon 140562620864256)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,663 DEBUG (jsonrpc/3) [Executor] FINISH thread <Thread(jsonrpc/3, started daemon 140562124498688)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,663 DEBUG (jsonrpc/1) [Executor] FINISH thread <Thread(jsonrpc/1, started daemon 140562132891392)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,664 DEBUG (jsonrpc/4) [Executor] FINISH thread <Thread(jsonrpc/4, started daemon 140562116105984)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,664 DEBUG (jsonrpc/0) [Executor] Worker stopped (executor:303)*14:19:02* 2018-06-27 14:18:09,665 DEBUG (jsonrpc/5) [Executor] FINISH thread <Thread(jsonrpc/5, started daemon 140562107713280)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,667 DEBUG (jsonrpc/7) [Executor] FINISH thread <Thread(jsonrpc/7, started daemon 140562090927872)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,667 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/0 (executor:290)*14:19:02* 2018-06-27 14:18:09,671 DEBUG (jsonrpc/0) [Executor] FINISH thread <Thread(jsonrpc/0, started daemon 140562612471552)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,673 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/1 (executor:290)*14:19:02* 2018-06-27 14:18:09,674 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/6 (executor:290)*14:19:02* 2018-Coverage.py warning: Module /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm was never imported. (module-not-imported)*14:19:05* pylint installdeps: pylint==1.8*14:19:22* pylint installed: The directory '/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.,astroid==1.6.5,backports.functools-lru-cache==1.5,backports.ssl-match-hostname==3.5.0.1,blivet==0.61.15.69,chardet==2.2.1,configparser==3.5.0,coverage==4.4.1,decorator==3.4.0,enum34==1.1.6,funcsigs==1.0.2,futures==3.2.0,iniparse==0.4,ioprocess==1.1.0,ipaddress==1.0.16,IPy==0.75,isort==4.3.4,kitchen==1.1.1,lazy-object-proxy==1.3.1,libvirt-python==3.9.0,Magic-file-extensions==0.2,mccabe==0.6.1,mock==2.0.0,netaddr==0.7.18,ovirt-imageio-common==1.4.1,pbr==3.1.1,pluggy==0.6.0,policycoreutils-default-encoding==0.1,pthreading==0.1.3,py==1.5.4,pycurl==7.19.0,pygpgme==0.3,pyinotify==0.9.4,pykickstart==1.99.66.18,pyliblzma==0.5.3,pylint==1.8.0,pyparted==3.9,python-augeas==0.5.0,python-dateutil==2.4.2,pyudev==0.15,pyxattr==0.5.1,PyYAML==3.10,requests==2.6.0,sanlock-python==3.6.0,seobject==0.1,sepolicy==1.1,singledispatch==3.4.0.3,six==1.11.0,subprocess32==3.2.6,tox==2.9.1,urlgrabber==3.10,urllib3==1.10.2,virtualenv==16.0.0,WebOb==1.2.3,wrapt==1.10.11,yum-metadata-parser==1.1.4*14:19:22* pylint runtests: PYTHONHASHSEED='528910716'*14:19:22* pylint runtests: commands[0] | pylint --errors-only static/usr/share/vdsm/sitecustomize.py lib/vdsm lib/vdsmclient lib/yajsonrpc*14:19:23* Problem importing module base.pyc: cannot import name get_node_last_lineno*14:19:24* Problem importing module base.py: cannot import name get_node_last_lineno*14:19:34* 06-27 14:18:09,674 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/7 (executor:290)*14:19:34* 2018-06-27 14:18:09,675 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/5 (executor:290)*14:19:34* 2018-06-27 14:18:09,675 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/2 (executor:290)*14:19:34* 2018-06-27 14:18:09,676 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/3 (executor:290)*14:19:34* 2018-06-27 14:18:09,676 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/4 (executor:290)*14:19:34* 2018-06-27 14:18:09,677 DEBUG (MainThread) [Scheduler] Stopping scheduler test.Scheduler (schedule:110)*14:19:34* 2018-06-27 14:18:09,678 DEBUG (test.Scheduler) [Scheduler] stopped (schedule:143)*14:19:34* 2018-06-27 14:18:09,679 DEBUG (test.Scheduler) [Scheduler] FINISH thread <Thread(test.Scheduler, started daemon 140562082535168)> (concurrent:196)*14:19:34* --------------------- >> end captured logging << --------------------- > > >

Ok, I'll take a look. śr., 27 cze 2018, 17:36 użytkownik Nir Soffer <nsoffer(a)redhat.com&gt; napisał: > On Wed, Jun 27, 2018 at 6:13 PM Piotr Kliczewski <pkliczew(a)redhat.com&gt; > wrote: > >> On Wed, Jun 27, 2018 at 5:01 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >> >>> This tests used to fail in the past, but since we fixed it or the code >>> >>> it never failed. >>> >>> >>> Maybe the slave was overloaded? >>> >>> >> Very possible. Can you paste a link to the job which failed? >> > > Here: https://jenkins.ovirt.org/job/vdsm_master_check- > patch-el7-x86_64/24130/ > > The next build passed. > > Maybe we need to solve the flakiness of some tests by running a flaky test > again, and let the build fail only if a test failed twice. I wonder if > there is some > pytest plugin doing this. > > >> >> >>> >>> *14:19:02* ======================================================================*14:19:02* ERROR: >>> >>> *14:19:02* ----------------------------------------------------------------------*14:19:02* Traceback (most recent call last):*14:19:02* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/testlib.py", line 143, in wrapper*14:19:02* return f(self, *args)*14:19:02* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/stomp_test.py", line 95, in test_echo*14:19:02* str(uuid4())),*14:19:02* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/lib/yajsonrpc/jsonrpcclient.py", line 77, in callMethod*14:19:02* raise exception.JsonRpcNoResponseError(method=methodName)*14:19:02* JsonRpcNoResponseError: No response for JSON-RPC request: {'method': 'echo'}*14:19:02* -------------------- >> begin captured logging << --------------------*14:19:02* 2018-06-27 14:17:54,524 DEBUG (MainThread) [vds.MultiProtocolAcceptor] Creating socket (host='::1', port=0, family=10, socketype=1, proto=6) (protocoldetector:225)*14:19:02* 2018-06-27 14:17:54,526 INFO (MainThread) [vds.MultiProtocolAcceptor] Listening at ::1:36713 (protocoldetector:183)*14:19:02* 2018-06-27 14:17:54,535 DEBUG (MainThread) [Scheduler] Starting scheduler test.Scheduler (schedule:98)*14:19:02* 2018-06-27 14:17:54,537 DEBUG (test.Scheduler) [Scheduler] START thread <Thread(test.Scheduler, started daemon 140562082535168)> (func=<bound method Scheduler._run of <vdsm.schedule.Scheduler object at 0x7fd74ca00390>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,538 DEBUG (test.Scheduler) [Scheduler] started (schedule:140)*14:19:02* 2018-06-27 14:17:54,546 DEBUG (JsonRpc (StompReactor)) [root] START thread <Thread(JsonRpc (StompReactor), started daemon 140562629256960)> (func=<bound method StompReactor.process_requests of <yajsonrpc.stompserver.StompReactor object at 0x7fd74ca006d0>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,547 DEBUG (MainThread) [Executor] Starting executor (executor:128)*14:19:02* 2018-06-27 14:17:54,549 DEBUG (MainThread) [Executor] Starting worker jsonrpc/0 (executor:286)*14:19:02* 2018-06-27 14:17:54,553 DEBUG (jsonrpc/0) [Executor] START thread <Thread(jsonrpc/0, started daemon 140562612471552)> (func=<bound method _Worker._run of <Worker name=jsonrpc/0 waiting task#=0 at 0x7fd74ca00650>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,554 DEBUG (jsonrpc/0) [Executor] Worker started (executor:298)*14:19:02* 2018-06-27 14:17:54,557 DEBUG (MainThread) [Executor] Starting worker jsonrpc/1 (executor:286)*14:19:02* 2018-06-27 14:17:54,558 DEBUG (MainThread) [Executor] Starting worker jsonrpc/2 (executor:286)*14:19:02* 2018-06-27 14:17:54,559 DEBUG (jsonrpc/2) [Executor] START thread <Thread(jsonrpc/2, started daemon 140562620864256)> (func=<bound method _Worker._run of <Worker name=jsonrpc/2 waiting task#=0 at 0x7fd74c9fb350>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,560 DEBUG (jsonrpc/2) [Executor] Worker started (executor:298)*14:19:02* 2018-06-27 14:17:54,561 DEBUG (MainThread) [Executor] Starting worker jsonrpc/3 (executor:286)*14:19:02* 2018-06-27 14:17:54,562 DEBUG (jsonrpc/3) [Executor] START thread <Thread(jsonrpc/3, started daemon 140562124498688)> (func=<bound method _Worker._run of <Worker name=jsonrpc/3 waiting task#=0 at 0x7fd74bc80290>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,563 DEBUG (jsonrpc/3) [Executor] Worker started (executor:298)*14:19:02* 2018-06-27 14:17:54,564 DEBUG (MainThread) [Executor] Starting worker jsonrpc/4 (executor:286)*14:19:02* 2018-06-27 14:17:54,565 DEBUG (jsonrpc/4) [Executor] START thread <Thread(jsonrpc/4, started daemon 140562116105984)> (func=<bound method _Worker._run of <Worker name=jsonrpc/4 waiting task#=0 at 0x7fd74bc80790>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,566 DEBUG (jsonrpc/4) [Executor] Worker started (executor:298)*14:19:02* 2018-06-27 14:17:54,568 DEBUG (jsonrpc/1) [Executor] START thread <Thread(jsonrpc/1, started daemon 140562132891392)> (func=<bound method _Worker._run of <Worker name=jsonrpc/1 waiting task#=0 at 0x7fd74c9fb690>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,569 DEBUG (jsonrpc/1) [Executor] Worker started (executor:298)*14:19:02* 2018-06-27 14:17:54,569 DEBUG (MainThread) [Executor] Starting worker jsonrpc/5 (executor:286)*14:19:02* 2018-06-27 14:17:54,570 DEBUG (jsonrpc/5) [Executor] START thread <Thread(jsonrpc/5, started daemon 140562107713280)> (func=<bound method _Worker._run of <Worker name=jsonrpc/5 waiting task#=0 at 0x7fd74bc80090>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,571 DEBUG (jsonrpc/5) [Executor] Worker started (executor:298)*14:19:02* 2018-06-27 14:17:54,572 DEBUG (MainThread) [Executor] Starting worker jsonrpc/6 (executor:286)*14:19:02* 2018-06-27 14:17:54,580 DEBUG (MainThread) [Executor] Starting worker jsonrpc/7 (executor:286)*14:19:02* 2018-06-27 14:17:54,603 DEBUG (jsonrpc/6) [Executor] START thread <Thread(jsonrpc/6, started daemon 140562099320576)> (func=<bound method _Worker._run of <Worker name=jsonrpc/6 waiting task#=0 at 0x7fd74d0dacd0>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,604 DEBUG (jsonrpc/6) [Executor] Worker started (executor:298)*14:19:02* 2018-06-27 14:17:54,615 DEBUG (jsonrpc/7) [Executor] START thread <Thread(jsonrpc/7, started daemon 140562090927872)> (func=<bound method _Worker._run of <Worker name=jsonrpc/7 waiting task#=0 at 0x7fd74bfe0310>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,616 DEBUG (JsonRpcServer) [root] START thread <Thread(JsonRpcServer, started daemon 140561730238208)> (func=<bound method JsonRpcServer.serve_requests of <yajsonrpc.JsonRpcServer object at 0x7fd74ca00690>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,618 DEBUG (MainThread) [vds.MultiProtocolAcceptor] Adding detector <yajsonrpc.stompserver.StompDetector instance at 0x7fd74c28cc68> (protocoldetector:210)*14:19:02* 2018-06-27 14:17:54,625 INFO (Detector thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:52432 (protocoldetector:61)*14:19:02* 2018-06-27 14:17:54,626 DEBUG (Detector thread) [ProtocolDetector.Detector] Using required_size=11 (protocoldetector:89)*14:19:02* 2018-06-27 14:17:54,627 DEBUG (jsonrpc/7) [Executor] Worker started (executor:298)*14:19:02* 2018-06-27 14:17:54,634 DEBUG (MainThread) [jsonrpc.AsyncoreClient] Sending response (stompclient:293)*14:19:02* 2018-06-27 14:17:54,634 DEBUG (Client ::1:36713) [root] START thread <Thread(Client ::1:36713, started daemon 140561713452800)> (func=<bound method Reactor.process_requests of <yajsonrpc.betterAsyncore.Reactor object at 0x7fd74bdb1d90>>, args=(), kwargs={}) (concurrent:193)*14:19:02* 2018-06-27 14:17:54,641 INFO (Detector thread) [ProtocolDetector.Detector] Detected protocol stomp from ::1:52432 (protocoldetector:125)*14:19:02* 2018-06-27 14:17:54,645 INFO (Detector thread) [Broker.StompAdapter] Processing CONNECT request (stompserver:94)*14:19:02* 2018-06-27 14:17:54,648 DEBUG (Client ::1:36713) [yajsonrpc.protocols.stomp.AsyncClient] Stomp connection established (stompclient:137)*14:19:02* 2018-06-27 14:17:54,651 DEBUG (jsonrpc/0) [jsonrpc.JsonRpcServer] Calling 'echo' in bridge with [u'Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut en'] (__init__:328)*14:19:02* 2018-06-27 14:17:54,652 DEBUG (jsonrpc/0) [jsonrpc.JsonRpcServer] Return 'echo' in bridge with Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut en (__init__:355)*14:19:02* 2018-06-27 14:17:54,653 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call echo succeeded in 0.01 seconds (__init__:311)*14:19:02* 2018-06-27 14:17:54,661 INFO (Detector thread) [Broker.StompAdapter] Subscribe command received (stompserver:123)*14:19:02* 2018-06-27 14:17:54,662 DEBUG (Detector thread) [protocoldetector.StompDetector] Stomp detected from ('::1', 52432) (stompserver:412)*14:19:02* 2018-06-27 14:18:09,649 DEBUG (MainThread) [vds.MultiProtocolAcceptor] Stopping Acceptor (protocoldetector:214)*14:19:02* 2018-06-27 14:18:09,650 INFO (MainThread) [jsonrpc.JsonRpcServer] Stopping JsonRPC Server (__init__:441)*14:19:02* 2018-06-27 14:18:09,651 DEBUG (MainThread) [Executor] Stopping executor (executor:137)*14:19:02* 2018-06-27 14:18:09,652 DEBUG (Client ::1:36713) [root] FINISH thread <Thread(Client ::1:36713, started daemon 140561713452800)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,652 DEBUG (JsonRpcServer) [root] FINISH thread <Thread(JsonRpcServer, started daemon 140561730238208)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,654 DEBUG (JsonRpc (StompReactor)) [root] FINISH thread <Thread(JsonRpc (StompReactor), started daemon 140562629256960)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,655 DEBUG (jsonrpc/2) [Executor] Worker stopped (executor:303)*14:19:02* 2018-06-27 14:18:09,658 DEBUG (jsonrpc/3) [Executor] Worker stopped (executor:303)*14:19:02* 2018-06-27 14:18:09,659 DEBUG (jsonrpc/4) [Executor] Worker stopped (executor:303)*14:19:02* 2018-06-27 14:18:09,660 DEBUG (jsonrpc/1) [Executor] Worker stopped (executor:303)*14:19:02* 2018-06-27 14:18:09,660 DEBUG (jsonrpc/5) [Executor] Worker stopped (executor:303)*14:19:02* 2018-06-27 14:18:09,660 DEBUG (jsonrpc/6) [Executor] Worker stopped (executor:303)*14:19:02* 2018-06-27 14:18:09,665 DEBUG (jsonrpc/6) [Executor] FINISH thread <Thread(jsonrpc/6, started daemon 140562099320576)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,661 DEBUG (jsonrpc/7) [Executor] Worker stopped (executor:303)*14:19:02* 2018-06-27 14:18:09,662 DEBUG (jsonrpc/2) [Executor] FINISH thread <Thread(jsonrpc/2, started daemon 140562620864256)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,663 DEBUG (jsonrpc/3) [Executor] FINISH thread <Thread(jsonrpc/3, started daemon 140562124498688)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,663 DEBUG (jsonrpc/1) [Executor] FINISH thread <Thread(jsonrpc/1, started daemon 140562132891392)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,664 DEBUG (jsonrpc/4) [Executor] FINISH thread <Thread(jsonrpc/4, started daemon 140562116105984)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,664 DEBUG (jsonrpc/0) [Executor] Worker stopped (executor:303)*14:19:02* 2018-06-27 14:18:09,665 DEBUG (jsonrpc/5) [Executor] FINISH thread <Thread(jsonrpc/5, started daemon 140562107713280)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,667 DEBUG (jsonrpc/7) [Executor] FINISH thread <Thread(jsonrpc/7, started daemon 140562090927872)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,667 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/0 (executor:290)*14:19:02* 2018-06-27 14:18:09,671 DEBUG (jsonrpc/0) [Executor] FINISH thread <Thread(jsonrpc/0, started daemon 140562612471552)> (concurrent:196)*14:19:02* 2018-06-27 14:18:09,673 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/1 (executor:290)*14:19:02* 2018-06-27 14:18:09,674 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/6 (executor:290)*14:19:02* 2018-Coverage.py warning: Module /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm was never imported. (module-not-imported)*14:19:05* pylint installdeps: pylint==1.8*14:19:22* pylint installed: The directory '/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.,astroid==1.6.5,backports.functools-lru-cache==1.5,backports.ssl-match-hostname==3.5.0.1,blivet==0.61.15.69,chardet==2.2.1,configparser==3.5.0,coverage==4.4.1,decorator==3.4.0,enum34==1.1.6,funcsigs==1.0.2,futures==3.2.0,iniparse==0.4,ioprocess==1.1.0,ipaddress==1.0.16,IPy==0.75,isort==4.3.4,kitchen==1.1.1,lazy-object-proxy==1.3.1,libvirt-python==3.9.0,Magic-file-extensions==0.2,mccabe==0.6.1,mock==2.0.0,netaddr==0.7.18,ovirt-imageio-common==1.4.1,pbr==3.1.1,pluggy==0.6.0,policycoreutils-default-encoding==0.1,pthreading==0.1.3,py==1.5.4,pycurl==7.19.0,pygpgme==0.3,pyinotify==0.9.4,pykickstart==1.99.66.18,pyliblzma==0.5.3,pylint==1.8.0,pyparted==3.9,python-augeas==0.5.0,python-dateutil==2.4.2,pyudev==0.15,pyxattr==0.5.1,PyYAML==3.10,requests==2.6.0,sanlock-python==3.6.0,seobject==0.1,sepolicy==1.1,singledispatch==3.4.0.3,six==1.11.0,subprocess32==3.2.6,tox==2.9.1,urlgrabber==3.10,urllib3==1.10.2,virtualenv==16.0.0,WebOb==1.2.3,wrapt==1.10.11,yum-metadata-parser==1.1.4*14:19:22* pylint runtests: PYTHONHASHSEED='528910716'*14:19:22* pylint runtests: commands[0] | pylint --errors-only static/usr/share/vdsm/sitecustomize.py lib/vdsm lib/vdsmclient lib/yajsonrpc*14:19:23* Problem importing module base.pyc: cannot import name get_node_last_lineno*14:19:24* Problem importing module base.py: cannot import name get_node_last_lineno*14:19:34* 06-27 14:18:09,674 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/7 (executor:290)*14:19:34* 2018-06-27 14:18:09,675 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/5 (executor:290)*14:19:34* 2018-06-27 14:18:09,675 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/2 (executor:290)*14:19:34* 2018-06-27 14:18:09,676 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/3 (executor:290)*14:19:34* 2018-06-27 14:18:09,676 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/4 (executor:290)*14:19:34* 2018-06-27 14:18:09,677 DEBUG (MainThread) [Scheduler] Stopping scheduler test.Scheduler (schedule:110)*14:19:34* 2018-06-27 14:18:09,678 DEBUG (test.Scheduler) [Scheduler] stopped (schedule:143)*14:19:34* 2018-06-27 14:18:09,679 DEBUG (test.Scheduler) [Scheduler] FINISH thread <Thread(test.Scheduler, started daemon 140562082535168)> (concurrent:196)*14:19:34* --------------------- >> end captured logging << --------------------- >>> >>> >>>

Here is the patch [1] which should fix it. [1] https://gerrit.ovirt.org/#/c/92690/ On Thu, Jun 28, 2018 at 2:44 PM Piotr Kliczewski <pkliczew(a)redhat.com&gt; wrote: > > Nir, > > It looks like we have a race condition where request arrive sooner than subscribe frame: > > 16:19:02 2018-06-27 14:17:54,653 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call echo succeeded in 0.01 seconds (__init__:311) > 16:19:02 2018-06-27 14:17:54,661 INFO (Detector thread) [Broker.StompAdapter] Subscribe command received (stompserver:123) > > Thanks for reporting I will push a patch to fix it. > > Thanks, > Piotr > > > On Wed, Jun 27, 2018 at 5:40 PM, Piotr Kliczewski <pkliczew(a)redhat.com&gt; wrote: >> >> Ok, I'll take a look. >> >> śr., 27 cze 2018, 17:36 użytkownik Nir Soffer <nsoffer(a)redhat.com&gt; napisał: >>> >>> On Wed, Jun 27, 2018 at 6:13 PM Piotr Kliczewski <pkliczew(a)redhat.com&gt; wrote: >>>> >>>> On Wed, Jun 27, 2018 at 5:01 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >>>>> >>>>> This tests used to fail in the past, but since we fixed it or the code >>>>> >>>>> it never failed. >>>>> >>>>> >>>>> Maybe the slave was overloaded? >>>> >>>> >>>> Very possible. Can you paste a link to the job which failed? >>> >>> >>> Here: https://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/24130/ >>> >>> The next build passed. >>> >>> Maybe we need to solve the flakiness of some tests by running a flaky test >>> again, and let the build fail only if a test failed twice. I wonder if there is some >>> pytest plugin doing this. >>> >>>> >>>> >>>>> >>>>> >>>>> 14:19:02 ====================================================================== >>>>> 14:19:02 ERROR: >>>>> >>>>> 14:19:02 ---------------------------------------------------------------------- >>>>> 14:19:02 Traceback (most recent call last): >>>>> 14:19:02 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/testlib.py", line 143, in wrapper >>>>> 14:19:02 return f(self, *args) >>>>> 14:19:02 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/stomp_test.py", line 95, in test_echo >>>>> 14:19:02 str(uuid4())), >>>>> 14:19:02 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/lib/yajsonrpc/jsonrpcclient.py", line 77, in callMethod >>>>> 14:19:02 raise exception.JsonRpcNoResponseError(method=methodName) >>>>> 14:19:02 JsonRpcNoResponseError: No response for JSON-RPC request: {'method': 'echo'} >>>>> 14:19:02 -------------------- >> begin captured logging << -------------------- >>>>> 14:19:02 2018-06-27 14:17:54,524 DEBUG (MainThread) [vds.MultiProtocolAcceptor] Creating socket (host='::1', port=0, family=10, socketype=1, proto=6) (protocoldetector:225) >>>>> 14:19:02 2018-06-27 14:17:54,526 INFO (MainThread) [vds.MultiProtocolAcceptor] Listening at ::1:36713 (protocoldetector:183) >>>>> 14:19:02 2018-06-27 14:17:54,535 DEBUG (MainThread) [Scheduler] Starting scheduler test.Scheduler (schedule:98) >>>>> 14:19:02 2018-06-27 14:17:54,537 DEBUG (test.Scheduler) [Scheduler] START thread <Thread(test.Scheduler, started daemon 140562082535168)> (func=<bound method Scheduler._run of <vdsm.schedule.Scheduler object at 0x7fd74ca00390>>, args=(), kwargs={}) (concurrent:193) >>>>> 14:19:02 2018-06-27 14:17:54,538 DEBUG (test.Scheduler) [Scheduler] started (schedule:140) >>>>> 14:19:02 2018-06-27 14:17:54,546 DEBUG (JsonRpc (StompReactor)) [root] START thread <Thread(JsonRpc (StompReactor), started daemon 140562629256960)> (func=<bound method StompReactor.process_requests of <yajsonrpc.stompserver.StompReactor object at 0x7fd74ca006d0>>, args=(), kwargs={}) (concurrent:193) >>>>> 14:19:02 2018-06-27 14:17:54,547 DEBUG (MainThread) [Executor] Starting executor (executor:128) >>>>> 14:19:02 2018-06-27 14:17:54,549 DEBUG (MainThread) [Executor] Starting worker jsonrpc/0 (executor:286) >>>>> 14:19:02 2018-06-27 14:17:54,553 DEBUG (jsonrpc/0) [Executor] START thread <Thread(jsonrpc/0, started daemon 140562612471552)> (func=<bound method _Worker._run of <Worker name=jsonrpc/0 waiting task#=0 at 0x7fd74ca00650>>, args=(), kwargs={}) (concurrent:193) >>>>> 14:19:02 2018-06-27 14:17:54,554 DEBUG (jsonrpc/0) [Executor] Worker started (executor:298) >>>>> 14:19:02 2018-06-27 14:17:54,557 DEBUG (MainThread) [Executor] Starting worker jsonrpc/1 (executor:286) >>>>> 14:19:02 2018-06-27 14:17:54,558 DEBUG (MainThread) [Executor] Starting worker jsonrpc/2 (executor:286) >>>>> 14:19:02 2018-06-27 14:17:54,559 DEBUG (jsonrpc/2) [Executor] START thread <Thread(jsonrpc/2, started daemon 140562620864256)> (func=<bound method _Worker._run of <Worker name=jsonrpc/2 waiting task#=0 at 0x7fd74c9fb350>>, args=(), kwargs={}) (concurrent:193) >>>>> 14:19:02 2018-06-27 14:17:54,560 DEBUG (jsonrpc/2) [Executor] Worker started (executor:298) >>>>> 14:19:02 2018-06-27 14:17:54,561 DEBUG (MainThread) [Executor] Starting worker jsonrpc/3 (executor:286) >>>>> 14:19:02 2018-06-27 14:17:54,562 DEBUG (jsonrpc/3) [Executor] START thread <Thread(jsonrpc/3, started daemon 140562124498688)> (func=<bound method _Worker._run of <Worker name=jsonrpc/3 waiting task#=0 at 0x7fd74bc80290>>, args=(), kwargs={}) (concurrent:193) >>>>> 14:19:02 2018-06-27 14:17:54,563 DEBUG (jsonrpc/3) [Executor] Worker started (executor:298) >>>>> 14:19:02 2018-06-27 14:17:54,564 DEBUG (MainThread) [Executor] Starting worker jsonrpc/4 (executor:286) >>>>> 14:19:02 2018-06-27 14:17:54,565 DEBUG (jsonrpc/4) [Executor] START thread <Thread(jsonrpc/4, started daemon 140562116105984)> (func=<bound method _Worker._run of <Worker name=jsonrpc/4 waiting task#=0 at 0x7fd74bc80790>>, args=(), kwargs={}) (concurrent:193) >>>>> 14:19:02 2018-06-27 14:17:54,566 DEBUG (jsonrpc/4) [Executor] Worker started (executor:298) >>>>> 14:19:02 2018-06-27 14:17:54,568 DEBUG (jsonrpc/1) [Executor] START thread <Thread(jsonrpc/1, started daemon 140562132891392)> (func=<bound method _Worker._run of <Worker name=jsonrpc/1 waiting task#=0 at 0x7fd74c9fb690>>, args=(), kwargs={}) (concurrent:193) >>>>> 14:19:02 2018-06-27 14:17:54,569 DEBUG (jsonrpc/1) [Executor] Worker started (executor:298) >>>>> 14:19:02 2018-06-27 14:17:54,569 DEBUG (MainThread) [Executor] Starting worker jsonrpc/5 (executor:286) >>>>> 14:19:02 2018-06-27 14:17:54,570 DEBUG (jsonrpc/5) [Executor] START thread <Thread(jsonrpc/5, started daemon 140562107713280)> (func=<bound method _Worker._run of <Worker name=jsonrpc/5 waiting task#=0 at 0x7fd74bc80090>>, args=(), kwargs={}) (concurrent:193) >>>>> 14:19:02 2018-06-27 14:17:54,571 DEBUG (jsonrpc/5) [Executor] Worker started (executor:298) >>>>> 14:19:02 2018-06-27 14:17:54,572 DEBUG (MainThread) [Executor] Starting worker jsonrpc/6 (executor:286) >>>>> 14:19:02 2018-06-27 14:17:54,580 DEBUG (MainThread) [Executor] Starting worker jsonrpc/7 (executor:286) >>>>> 14:19:02 2018-06-27 14:17:54,603 DEBUG (jsonrpc/6) [Executor] START thread <Thread(jsonrpc/6, started daemon 140562099320576)> (func=<bound method _Worker._run of <Worker name=jsonrpc/6 waiting task#=0 at 0x7fd74d0dacd0>>, args=(), kwargs={}) (concurrent:193) >>>>> 14:19:02 2018-06-27 14:17:54,604 DEBUG (jsonrpc/6) [Executor] Worker started (executor:298) >>>>> 14:19:02 2018-06-27 14:17:54,615 DEBUG (jsonrpc/7) [Executor] START thread <Thread(jsonrpc/7, started daemon 140562090927872)> (func=<bound method _Worker._run of <Worker name=jsonrpc/7 waiting task#=0 at 0x7fd74bfe0310>>, args=(), kwargs={}) (concurrent:193) >>>>> 14:19:02 2018-06-27 14:17:54,616 DEBUG (JsonRpcServer) [root] START thread <Thread(JsonRpcServer, started daemon 140561730238208)> (func=<bound method JsonRpcServer.serve_requests of <yajsonrpc.JsonRpcServer object at 0x7fd74ca00690>>, args=(), kwargs={}) (concurrent:193) >>>>> 14:19:02 2018-06-27 14:17:54,618 DEBUG (MainThread) [vds.MultiProtocolAcceptor] Adding detector <yajsonrpc.stompserver.StompDetector instance at 0x7fd74c28cc68> (protocoldetector:210) >>>>> 14:19:02 2018-06-27 14:17:54,625 INFO (Detector thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:52432 (protocoldetector:61) >>>>> 14:19:02 2018-06-27 14:17:54,626 DEBUG (Detector thread) [ProtocolDetector.Detector] Using required_size=11 (protocoldetector:89) >>>>> 14:19:02 2018-06-27 14:17:54,627 DEBUG (jsonrpc/7) [Executor] Worker started (executor:298) >>>>> 14:19:02 2018-06-27 14:17:54,634 DEBUG (MainThread) [jsonrpc.AsyncoreClient] Sending response (stompclient:293) >>>>> 14:19:02 2018-06-27 14:17:54,634 DEBUG (Client ::1:36713) [root] START thread <Thread(Client ::1:36713, started daemon 140561713452800)> (func=<bound method Reactor.process_requests of <yajsonrpc.betterAsyncore.Reactor object at 0x7fd74bdb1d90>>, args=(), kwargs={}) (concurrent:193) >>>>> 14:19:02 2018-06-27 14:17:54,641 INFO (Detector thread) [ProtocolDetector.Detector] Detected protocol stomp from ::1:52432 (protocoldetector:125) >>>>> 14:19:02 2018-06-27 14:17:54,645 INFO (Detector thread) [Broker.StompAdapter] Processing CONNECT request (stompserver:94) >>>>> 14:19:02 2018-06-27 14:17:54,648 DEBUG (Client ::1:36713) [yajsonrpc.protocols.stomp.AsyncClient] Stomp connection established (stompclient:137) >>>>> 14:19:02 2018-06-27 14:17:54,651 DEBUG (jsonrpc/0) [jsonrpc.JsonRpcServer] Calling 'echo' in bridge with [u'Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut en'] (__init__:328) >>>>> 14:19:02 2018-06-27 14:17:54,652 DEBUG (jsonrpc/0) [jsonrpc.JsonRpcServer] Return 'echo' in bridge with Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut en (__init__:355) >>>>> 14:19:02 2018-06-27 14:17:54,653 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call echo succeeded in 0.01 seconds (__init__:311) >>>>> 14:19:02 2018-06-27 14:17:54,661 INFO (Detector thread) [Broker.StompAdapter] Subscribe command received (stompserver:123) >>>>> 14:19:02 2018-06-27 14:17:54,662 DEBUG (Detector thread) [protocoldetector.StompDetector] Stomp detected from ('::1', 52432) (stompserver:412) >>>>> 14:19:02 2018-06-27 14:18:09,649 DEBUG (MainThread) [vds.MultiProtocolAcceptor] Stopping Acceptor (protocoldetector:214) >>>>> 14:19:02 2018-06-27 14:18:09,650 INFO (MainThread) [jsonrpc.JsonRpcServer] Stopping JsonRPC Server (__init__:441) >>>>> 14:19:02 2018-06-27 14:18:09,651 DEBUG (MainThread) [Executor] Stopping executor (executor:137) >>>>> 14:19:02 2018-06-27 14:18:09,652 DEBUG (Client ::1:36713) [root] FINISH thread <Thread(Client ::1:36713, started daemon 140561713452800)> (concurrent:196) >>>>> 14:19:02 2018-06-27 14:18:09,652 DEBUG (JsonRpcServer) [root] FINISH thread <Thread(JsonRpcServer, started daemon 140561730238208)> (concurrent:196) >>>>> 14:19:02 2018-06-27 14:18:09,654 DEBUG (JsonRpc (StompReactor)) [root] FINISH thread <Thread(JsonRpc (StompReactor), started daemon 140562629256960)> (concurrent:196) >>>>> 14:19:02 2018-06-27 14:18:09,655 DEBUG (jsonrpc/2) [Executor] Worker stopped (executor:303) >>>>> 14:19:02 2018-06-27 14:18:09,658 DEBUG (jsonrpc/3) [Executor] Worker stopped (executor:303) >>>>> 14:19:02 2018-06-27 14:18:09,659 DEBUG (jsonrpc/4) [Executor] Worker stopped (executor:303) >>>>> 14:19:02 2018-06-27 14:18:09,660 DEBUG (jsonrpc/1) [Executor] Worker stopped (executor:303) >>>>> 14:19:02 2018-06-27 14:18:09,660 DEBUG (jsonrpc/5) [Executor] Worker stopped (executor:303) >>>>> 14:19:02 2018-06-27 14:18:09,660 DEBUG (jsonrpc/6) [Executor] Worker stopped (executor:303) >>>>> 14:19:02 2018-06-27 14:18:09,665 DEBUG (jsonrpc/6) [Executor] FINISH thread <Thread(jsonrpc/6, started daemon 140562099320576)> (concurrent:196) >>>>> 14:19:02 2018-06-27 14:18:09,661 DEBUG (jsonrpc/7) [Executor] Worker stopped (executor:303) >>>>> 14:19:02 2018-06-27 14:18:09,662 DEBUG (jsonrpc/2) [Executor] FINISH thread <Thread(jsonrpc/2, started daemon 140562620864256)> (concurrent:196) >>>>> 14:19:02 2018-06-27 14:18:09,663 DEBUG (jsonrpc/3) [Executor] FINISH thread <Thread(jsonrpc/3, started daemon 140562124498688)> (concurrent:196) >>>>> 14:19:02 2018-06-27 14:18:09,663 DEBUG (jsonrpc/1) [Executor] FINISH thread <Thread(jsonrpc/1, started daemon 140562132891392)> (concurrent:196) >>>>> 14:19:02 2018-06-27 14:18:09,664 DEBUG (jsonrpc/4) [Executor] FINISH thread <Thread(jsonrpc/4, started daemon 140562116105984)> (concurrent:196) >>>>> 14:19:02 2018-06-27 14:18:09,664 DEBUG (jsonrpc/0) [Executor] Worker stopped (executor:303) >>>>> 14:19:02 2018-06-27 14:18:09,665 DEBUG (jsonrpc/5) [Executor] FINISH thread <Thread(jsonrpc/5, started daemon 140562107713280)> (concurrent:196) >>>>> 14:19:02 2018-06-27 14:18:09,667 DEBUG (jsonrpc/7) [Executor] FINISH thread <Thread(jsonrpc/7, started daemon 140562090927872)> (concurrent:196) >>>>> 14:19:02 2018-06-27 14:18:09,667 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/0 (executor:290) >>>>> 14:19:02 2018-06-27 14:18:09,671 DEBUG (jsonrpc/0) [Executor] FINISH thread <Thread(jsonrpc/0, started daemon 140562612471552)> (concurrent:196) >>>>> 14:19:02 2018-06-27 14:18:09,673 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/1 (executor:290) >>>>> 14:19:02 2018-06-27 14:18:09,674 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/6 (executor:290) >>>>> 14:19:02 2018-Coverage.py warning: Module /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm was never imported. (module-not-imported) >>>>> 14:19:05 pylint installdeps: pylint==1.8 >>>>> 14:19:22 pylint installed: The directory '/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.,astroid==1.6.5,backports.functools-lru-cache==1.5,backports.ssl-match-hostname==3.5.0.1,blivet==0.61.15.69,chardet==2.2.1,configparser==3.5.0,coverage==4.4.1,decorator==3.4.0,enum34==1.1.6,funcsigs==1.0.2,futures==3.2.0,iniparse==0.4,ioprocess==1.1.0,ipaddress==1.0.16,IPy==0.75,isort==4.3.4,kitchen==1.1.1,lazy-object-proxy==1.3.1,libvirt-python==3.9.0,Magic-file-extensions==0.2,mccabe==0.6.1,mock==2.0.0,netaddr==0.7.18,ovirt-imageio-common==1.4.1,pbr==3.1.1,pluggy==0.6.0,policycoreutils-default-encoding==0.1,pthreading==0.1.3,py==1.5.4,pycurl==7.19.0,pygpgme==0.3,pyinotify==0.9.4,pykickstart==1.99.66.18,pyliblzma==0.5.3,pylint==1.8.0,pyparted==3.9,python-augeas==0.5.0,python-dateutil==2.4.2,pyudev==0.15,pyxattr==0.5.1,PyYAML==3.10,requests==2.6.0,sanlock-python==3.6.0,seobject==0.1,sepolicy==1.1,singledispatch==3.4.0.3,six==1.11.0,subprocess32==3.2.6,tox==2.9.1,urlgrabber==3.10,urllib3==1.10.2,virtualenv==16.0.0,WebOb==1.2.3,wrapt==1.10.11,yum-metadata-parser==1.1.4 >>>>> 14:19:22 pylint runtests: PYTHONHASHSEED='528910716' >>>>> 14:19:22 pylint runtests: commands[0] | pylint --errors-only static/usr/share/vdsm/sitecustomize.py lib/vdsm lib/vdsmclient lib/yajsonrpc >>>>> 14:19:23 Problem importing module base.pyc: cannot import name get_node_last_lineno >>>>> 14:19:24 Problem importing module base.py: cannot import name get_node_last_lineno >>>>> 14:19:34 06-27 14:18:09,674 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/7 (executor:290) >>>>> 14:19:34 2018-06-27 14:18:09,675 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/5 (executor:290) >>>>> 14:19:34 2018-06-27 14:18:09,675 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/2 (executor:290) >>>>> 14:19:34 2018-06-27 14:18:09,676 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/3 (executor:290) >>>>> 14:19:34 2018-06-27 14:18:09,676 DEBUG (MainThread) [Executor] Waiting for worker jsonrpc/4 (executor:290) >>>>> 14:19:34 2018-06-27 14:18:09,677 DEBUG (MainThread) [Scheduler] Stopping scheduler test.Scheduler (schedule:110) >>>>> 14:19:34 2018-06-27 14:18:09,678 DEBUG (test.Scheduler) [Scheduler] stopped (schedule:143) >>>>> 14:19:34 2018-06-27 14:18:09,679 DEBUG (test.Scheduler) [Scheduler] FINISH thread <Thread(test.Scheduler, started daemon 140562082535168)> (concurrent:196) >>>>> 14:19:34 --------------------- >> end captured logging << --------------------- >>>>> >>>>> > > _______________________________________________ > Devel mailing list -- devel(a)ovirt.org > To unsubscribe send an email to devel-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ACPGZ4N3ULQ... _______________________________________________ Devel mailing list -- devel(a)ovirt.org To unsubscribe send an email to devel-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/IJJZC7GEB2J...

On Wed, Nov 7, 2018 at 5:21 AM Germano Veit Michel <germano(a)redhat.com&gt; wrote: > > On Sat, Jun 30, 2018 at 12:43 AM Piotr Kliczewski < > piotr.kliczewski(a)gmail.com&gt; wrote: > >> Here is the patch [1] which should fix it. >> >> [1] https://gerrit.ovirt.org/#/c/92690/ >> On Thu, Jun 28, 2018 at 2:44 PM Piotr Kliczewski <pkliczew(a)redhat.com&gt; >> wrote: >> > >> > Nir, >> > >> > It looks like we have a race condition where request arrive sooner >> than subscribe frame: >> > >> > 16:19:02 2018-06-27 14:17:54,653 INFO (jsonrpc/0) >> [jsonrpc.JsonRpcServer] RPC call echo succeeded in 0.01 seconds >> (__init__:311) >> > 16:19:02 2018-06-27 14:17:54,661 INFO (Detector thread) >> [Broker.StompAdapter] Subscribe command received (stompserver:123) >> > >> > Thanks for reporting I will push a patch to fix it. >> > >> > Thanks, >> > Piotr >> > >> > >> > On Wed, Jun 27, 2018 at 5:40 PM, Piotr Kliczewski <pkliczew(a)redhat.com&gt; >> wrote: >> >> >> >> Ok, I'll take a look. >> >> >> >> śr., 27 cze 2018, 17:36 użytkownik Nir Soffer <nsoffer(a)redhat.com&gt; >> napisał: >> >>> >> >>> On Wed, Jun 27, 2018 at 6:13 PM Piotr Kliczewski < >> pkliczew(a)redhat.com&gt; wrote: >> >>>> >> >>>> On Wed, Jun 27, 2018 at 5:01 PM, Nir Soffer <nsoffer(a)redhat.com&gt; >> wrote: >> >>>>> >> >>>>> This tests used to fail in the past, but since we fixed it or the >> code >> >>>>> >> >>>>> it never failed. >> >>>>> >> >>>>> >> >>>>> Maybe the slave was overloaded? >> >>>> >> >>>> >> >>>> Very possible. Can you paste a link to the job which failed? >> >>> >> >>> >> >>> Here: >> https://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/24130/ >> >>> >> >>> The next build passed. >> >>> >> >>> Maybe we need to solve the flakiness of some tests by running a >> flaky test >> >>> again, and let the build fail only if a test failed twice. I wonder >> if there is some >> >>> pytest plugin doing this. >> >>> >> >>>> >> >>>> >> >>>>> >> >>>>> >> >>>>> 14:19:02 >> ====================================================================== >> >>>>> 14:19:02 ERROR: >> >>>>> >> >>>>> 14:19:02 >> ---------------------------------------------------------------------- >> >>>>> 14:19:02 Traceback (most recent call last): >> >>>>> 14:19:02 File >> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/testlib.py", >> line 143, in wrapper >> >>>>> 14:19:02 return f(self, *args) >> >>>>> 14:19:02 File >> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/stomp_test.py", >> line 95, in test_echo >> >>>>> 14:19:02 str(uuid4())), >> >>>>> 14:19:02 File >> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/lib/yajsonrpc/jsonrpcclient.py", >> line 77, in callMethod >> >>>>> 14:19:02 raise >> exception.JsonRpcNoResponseError(method=methodName) >> >>>>> 14:19:02 JsonRpcNoResponseError: No response for JSON-RPC request: >> {'method': 'echo'} >> >>>>> 14:19:02 -------------------- >> begin captured logging << >> -------------------- >> >>>>> 14:19:02 2018-06-27 14:17:54,524 DEBUG (MainThread) >> [vds.MultiProtocolAcceptor] Creating socket (host='::1', port=0, family=10, >> socketype=1, proto=6) (protocoldetector:225) >> >>>>> 14:19:02 2018-06-27 14:17:54,526 INFO (MainThread) >> [vds.MultiProtocolAcceptor] Listening at ::1:36713 (protocoldetector:183) >> >>>>> 14:19:02 2018-06-27 14:17:54,535 DEBUG (MainThread) [Scheduler] >> Starting scheduler test.Scheduler (schedule:98) >> >>>>> 14:19:02 2018-06-27 14:17:54,537 DEBUG (test.Scheduler) >> [Scheduler] START thread <Thread(test.Scheduler, started daemon >> 140562082535168)> (func=<bound method Scheduler._run of >> <vdsm.schedule.Scheduler object at 0x7fd74ca00390>>, args=(), kwargs={}) >> (concurrent:193) >> >>>>> 14:19:02 2018-06-27 14:17:54,538 DEBUG (test.Scheduler) >> [Scheduler] started (schedule:140) >> >>>>> 14:19:02 2018-06-27 14:17:54,546 DEBUG (JsonRpc (StompReactor)) >> [root] START thread <Thread(JsonRpc (StompReactor), started daemon >> 140562629256960)> (func=<bound method StompReactor.process_requests of >> <yajsonrpc.stompserver.StompReactor object at 0x7fd74ca006d0>>, args=(), >> kwargs={}) (concurrent:193) >> >>>>> 14:19:02 2018-06-27 14:17:54,547 DEBUG (MainThread) [Executor] >> Starting executor (executor:128) >> >>>>> 14:19:02 2018-06-27 14:17:54,549 DEBUG (MainThread) [Executor] >> Starting worker jsonrpc/0 (executor:286) >> >>>>> 14:19:02 2018-06-27 14:17:54,553 DEBUG (jsonrpc/0) [Executor] >> START thread <Thread(jsonrpc/0, started daemon 140562612471552)> >> (func=<bound method _Worker._run of <Worker name=jsonrpc/0 waiting task#=0 >> at 0x7fd74ca00650>>, args=(), kwargs={}) (concurrent:193) >> >>>>> 14:19:02 2018-06-27 14:17:54,554 DEBUG (jsonrpc/0) [Executor] >> Worker started (executor:298) >> >>>>> 14:19:02 2018-06-27 14:17:54,557 DEBUG (MainThread) [Executor] >> Starting worker jsonrpc/1 (executor:286) >> >>>>> 14:19:02 2018-06-27 14:17:54,558 DEBUG (MainThread) [Executor] >> Starting worker jsonrpc/2 (executor:286) >> >>>>> 14:19:02 2018-06-27 14:17:54,559 DEBUG (jsonrpc/2) [Executor] >> START thread <Thread(jsonrpc/2, started daemon 140562620864256)> >> (func=<bound method _Worker._run of <Worker name=jsonrpc/2 waiting task#=0 >> at 0x7fd74c9fb350>>, args=(), kwargs={}) (concurrent:193) >> >>>>> 14:19:02 2018-06-27 14:17:54,560 DEBUG (jsonrpc/2) [Executor] >> Worker started (executor:298) >> >>>>> 14:19:02 2018-06-27 14:17:54,561 DEBUG (MainThread) [Executor] >> Starting worker jsonrpc/3 (executor:286) >> >>>>> 14:19:02 2018-06-27 14:17:54,562 DEBUG (jsonrpc/3) [Executor] >> START thread <Thread(jsonrpc/3, started daemon 140562124498688)> >> (func=<bound method _Worker._run of <Worker name=jsonrpc/3 waiting task#=0 >> at 0x7fd74bc80290>>, args=(), kwargs={}) (concurrent:193) >> >>>>> 14:19:02 2018-06-27 14:17:54,563 DEBUG (jsonrpc/3) [Executor] >> Worker started (executor:298) >> >>>>> 14:19:02 2018-06-27 14:17:54,564 DEBUG (MainThread) [Executor] >> Starting worker jsonrpc/4 (executor:286) >> >>>>> 14:19:02 2018-06-27 14:17:54,565 DEBUG (jsonrpc/4) [Executor] >> START thread <Thread(jsonrpc/4, started daemon 140562116105984)> >> (func=<bound method _Worker._run of <Worker name=jsonrpc/4 waiting task#=0 >> at 0x7fd74bc80790>>, args=(), kwargs={}) (concurrent:193) >> >>>>> 14:19:02 2018-06-27 14:17:54,566 DEBUG (jsonrpc/4) [Executor] >> Worker started (executor:298) >> >>>>> 14:19:02 2018-06-27 14:17:54,568 DEBUG (jsonrpc/1) [Executor] >> START thread <Thread(jsonrpc/1, started daemon 140562132891392)> >> (func=<bound method _Worker._run of <Worker name=jsonrpc/1 waiting task#=0 >> at 0x7fd74c9fb690>>, args=(), kwargs={}) (concurrent:193) >> >>>>> 14:19:02 2018-06-27 14:17:54,569 DEBUG (jsonrpc/1) [Executor] >> Worker started (executor:298) >> >>>>> 14:19:02 2018-06-27 14:17:54,569 DEBUG (MainThread) [Executor] >> Starting worker jsonrpc/5 (executor:286) >> >>>>> 14:19:02 2018-06-27 14:17:54,570 DEBUG (jsonrpc/5) [Executor] >> START thread <Thread(jsonrpc/5, started daemon 140562107713280)> >> (func=<bound method _Worker._run of <Worker name=jsonrpc/5 waiting task#=0 >> at 0x7fd74bc80090>>, args=(), kwargs={}) (concurrent:193) >> >>>>> 14:19:02 2018-06-27 14:17:54,571 DEBUG (jsonrpc/5) [Executor] >> Worker started (executor:298) >> >>>>> 14:19:02 2018-06-27 14:17:54,572 DEBUG (MainThread) [Executor] >> Starting worker jsonrpc/6 (executor:286) >> >>>>> 14:19:02 2018-06-27 14:17:54,580 DEBUG (MainThread) [Executor] >> Starting worker jsonrpc/7 (executor:286) >> >>>>> 14:19:02 2018-06-27 14:17:54,603 DEBUG (jsonrpc/6) [Executor] >> START thread <Thread(jsonrpc/6, started daemon 140562099320576)> >> (func=<bound method _Worker._run of <Worker name=jsonrpc/6 waiting task#=0 >> at 0x7fd74d0dacd0>>, args=(), kwargs={}) (concurrent:193) >> >>>>> 14:19:02 2018-06-27 14:17:54,604 DEBUG (jsonrpc/6) [Executor] >> Worker started (executor:298) >> >>>>> 14:19:02 2018-06-27 14:17:54,615 DEBUG (jsonrpc/7) [Executor] >> START thread <Thread(jsonrpc/7, started daemon 140562090927872)> >> (func=<bound method _Worker._run of <Worker name=jsonrpc/7 waiting task#=0 >> at 0x7fd74bfe0310>>, args=(), kwargs={}) (concurrent:193) >> >>>>> 14:19:02 2018-06-27 14:17:54,616 DEBUG (JsonRpcServer) [root] >> START thread <Thread(JsonRpcServer, started daemon 140561730238208)> >> (func=<bound method JsonRpcServer.serve_requests of >> <yajsonrpc.JsonRpcServer object at 0x7fd74ca00690>>, args=(), kwargs={}) >> (concurrent:193) >> >>>>> 14:19:02 2018-06-27 14:17:54,618 DEBUG (MainThread) >> [vds.MultiProtocolAcceptor] Adding detector >> <yajsonrpc.stompserver.StompDetector instance at 0x7fd74c28cc68> >> (protocoldetector:210) >> >>>>> 14:19:02 2018-06-27 14:17:54,625 INFO (Detector thread) >> [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:52432 >> (protocoldetector:61) >> >>>>> 14:19:02 2018-06-27 14:17:54,626 DEBUG (Detector thread) >> [ProtocolDetector.Detector] Using required_size=11 (protocoldetector:89) >> >>>>> 14:19:02 2018-06-27 14:17:54,627 DEBUG (jsonrpc/7) [Executor] >> Worker started (executor:298) >> >>>>> 14:19:02 2018-06-27 14:17:54,634 DEBUG (MainThread) >> [jsonrpc.AsyncoreClient] Sending response (stompclient:293) >> >>>>> 14:19:02 2018-06-27 14:17:54,634 DEBUG (Client ::1:36713) [root] >> START thread <Thread(Client ::1:36713, started daemon 140561713452800)> >> (func=<bound method Reactor.process_requests of >> <yajsonrpc.betterAsyncore.Reactor object at 0x7fd74bdb1d90>>, args=(), >> kwargs={}) (concurrent:193) >> >>>>> 14:19:02 2018-06-27 14:17:54,641 INFO (Detector thread) >> [ProtocolDetector.Detector] Detected protocol stomp from ::1:52432 >> (protocoldetector:125) >> >>>>> 14:19:02 2018-06-27 14:17:54,645 INFO (Detector thread) >> [Broker.StompAdapter] Processing CONNECT request (stompserver:94) >> >>>>> 14:19:02 2018-06-27 14:17:54,648 DEBUG (Client ::1:36713) >> [yajsonrpc.protocols.stomp.AsyncClient] Stomp connection established >> (stompclient:137) >> >>>>> 14:19:02 2018-06-27 14:17:54,651 DEBUG (jsonrpc/0) >> [jsonrpc.JsonRpcServer] Calling 'echo' in bridge with [u'Lorem ipsum dolor >> sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut >> labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud >> exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis >> aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu >> fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt >> in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor >> sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut >> labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud >> exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis >> aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu >> fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt >> in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor >> sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut >> labore et dolore magna aliqua. Ut en'] (__init__:328) >> >>>>> 14:19:02 2018-06-27 14:17:54,652 DEBUG (jsonrpc/0) >> [jsonrpc.JsonRpcServer] Return 'echo' in bridge with Lorem ipsum dolor sit >> amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut >> labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud >> exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis >> aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu >> fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt >> in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor >> sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut >> labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud >> exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis >> aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu >> fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt >> in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor >> sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut >> labore et dolore magna aliqua. Ut en (__init__:355) >> >>>>> 14:19:02 2018-06-27 14:17:54,653 INFO (jsonrpc/0) >> [jsonrpc.JsonRpcServer] RPC call echo succeeded in 0.01 seconds >> (__init__:311) >> >>>>> 14:19:02 2018-06-27 14:17:54,661 INFO (Detector thread) >> [Broker.StompAdapter] Subscribe command received (stompserver:123) >> >>>>> 14:19:02 2018-06-27 14:17:54,662 DEBUG (Detector thread) >> [protocoldetector.StompDetector] Stomp detected from ('::1', 52432) >> (stompserver:412) >> >>>>> 14:19:02 2018-06-27 14:18:09,649 DEBUG (MainThread) >> [vds.MultiProtocolAcceptor] Stopping Acceptor (protocoldetector:214) >> >>>>> 14:19:02 2018-06-27 14:18:09,650 INFO (MainThread) >> [jsonrpc.JsonRpcServer] Stopping JsonRPC Server (__init__:441) >> >>>>> 14:19:02 2018-06-27 14:18:09,651 DEBUG (MainThread) [Executor] >> Stopping executor (executor:137) >> >>>>> 14:19:02 2018-06-27 14:18:09,652 DEBUG (Client ::1:36713) [root] >> FINISH thread <Thread(Client ::1:36713, started daemon 140561713452800)> >> (concurrent:196) >> >>>>> 14:19:02 2018-06-27 14:18:09,652 DEBUG (JsonRpcServer) [root] >> FINISH thread <Thread(JsonRpcServer, started daemon 140561730238208)> >> (concurrent:196) >> >>>>> 14:19:02 2018-06-27 14:18:09,654 DEBUG (JsonRpc (StompReactor)) >> [root] FINISH thread <Thread(JsonRpc (StompReactor), started daemon >> 140562629256960)> (concurrent:196) >> >>>>> 14:19:02 2018-06-27 14:18:09,655 DEBUG (jsonrpc/2) [Executor] >> Worker stopped (executor:303) >> >>>>> 14:19:02 2018-06-27 14:18:09,658 DEBUG (jsonrpc/3) [Executor] >> Worker stopped (executor:303) >> >>>>> 14:19:02 2018-06-27 14:18:09,659 DEBUG (jsonrpc/4) [Executor] >> Worker stopped (executor:303) >> >>>>> 14:19:02 2018-06-27 14:18:09,660 DEBUG (jsonrpc/1) [Executor] >> Worker stopped (executor:303) >> >>>>> 14:19:02 2018-06-27 14:18:09,660 DEBUG (jsonrpc/5) [Executor] >> Worker stopped (executor:303) >> >>>>> 14:19:02 2018-06-27 14:18:09,660 DEBUG (jsonrpc/6) [Executor] >> Worker stopped (executor:303) >> >>>>> 14:19:02 2018-06-27 14:18:09,665 DEBUG (jsonrpc/6) [Executor] >> FINISH thread <Thread(jsonrpc/6, started daemon 140562099320576)> >> (concurrent:196) >> >>>>> 14:19:02 2018-06-27 14:18:09,661 DEBUG (jsonrpc/7) [Executor] >> Worker stopped (executor:303) >> >>>>> 14:19:02 2018-06-27 14:18:09,662 DEBUG (jsonrpc/2) [Executor] >> FINISH thread <Thread(jsonrpc/2, started daemon 140562620864256)> >> (concurrent:196) >> >>>>> 14:19:02 2018-06-27 14:18:09,663 DEBUG (jsonrpc/3) [Executor] >> FINISH thread <Thread(jsonrpc/3, started daemon 140562124498688)> >> (concurrent:196) >> >>>>> 14:19:02 2018-06-27 14:18:09,663 DEBUG (jsonrpc/1) [Executor] >> FINISH thread <Thread(jsonrpc/1, started daemon 140562132891392)> >> (concurrent:196) >> >>>>> 14:19:02 2018-06-27 14:18:09,664 DEBUG (jsonrpc/4) [Executor] >> FINISH thread <Thread(jsonrpc/4, started daemon 140562116105984)> >> (concurrent:196) >> >>>>> 14:19:02 2018-06-27 14:18:09,664 DEBUG (jsonrpc/0) [Executor] >> Worker stopped (executor:303) >> >>>>> 14:19:02 2018-06-27 14:18:09,665 DEBUG (jsonrpc/5) [Executor] >> FINISH thread <Thread(jsonrpc/5, started daemon 140562107713280)> >> (concurrent:196) >> >>>>> 14:19:02 2018-06-27 14:18:09,667 DEBUG (jsonrpc/7) [Executor] >> FINISH thread <Thread(jsonrpc/7, started daemon 140562090927872)> >> (concurrent:196) >> >>>>> 14:19:02 2018-06-27 14:18:09,667 DEBUG (MainThread) [Executor] >> Waiting for worker jsonrpc/0 (executor:290) >> >>>>> 14:19:02 2018-06-27 14:18:09,671 DEBUG (jsonrpc/0) [Executor] >> FINISH thread <Thread(jsonrpc/0, started daemon 140562612471552)> >> (concurrent:196) >> >>>>> 14:19:02 2018-06-27 14:18:09,673 DEBUG (MainThread) [Executor] >> Waiting for worker jsonrpc/1 (executor:290) >> >>>>> 14:19:02 2018-06-27 14:18:09,674 DEBUG (MainThread) [Executor] >> Waiting for worker jsonrpc/6 (executor:290) >> >>>>> 14:19:02 2018-Coverage.py warning: Module >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm was >> never imported. (module-not-imported) >> >>>>> 14:19:05 pylint installdeps: pylint==1.8 >> >>>>> 14:19:22 pylint installed: The directory >> '/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/.cache/pip/http' >> or its parent directory is not owned by the current user and the cache has >> been disabled. Please check the permissions and owner of that directory. If >> executing pip with sudo, you may want sudo's -H >> flag.,astroid==1.6.5,backports.functools-lru-cache==1.5,backports.ssl-match-hostname==3.5.0.1,blivet==0.61.15.69,chardet==2.2.1,configparser==3.5.0,coverage==4.4.1,decorator==3.4.0,enum34==1.1.6,funcsigs==1.0.2,futures==3.2.0,iniparse==0.4,ioprocess==1.1.0,ipaddress==1.0.16,IPy==0.75,isort==4.3.4,kitchen==1.1.1,lazy-object-proxy==1.3.1,libvirt-python==3.9.0,Magic-file-extensions==0.2,mccabe==0.6.1,mock==2.0.0,netaddr==0.7.18,ovirt-imageio-common==1.4.1,pbr==3.1.1,pluggy==0.6.0,policycoreutils-default-encoding==0.1,pthreading==0.1.3,py==1.5.4,pycurl==7.19.0,pygpgme==0.3,pyinotify==0.9.4,pykickstart==1.99.66.18,pyliblzma==0.5.3,pylint==1.8.0,pyparted==3.9,python-augeas==0.5.0,python-dateutil==2.4.2,pyudev==0.15,pyxattr==0.5.1,PyYAML==3.10,requests==2.6.0,sanlock-python==3.6.0,seobject==0.1,sepolicy==1.1,singledispatch==3.4.0.3,six==1.11.0,subprocess32==3.2.6,tox==2.9.1,urlgrabber==3.10,urllib3==1.10.2,virtualenv==16.0.0,WebOb==1.2.3,wrapt==1.10.11,yum-metadata-parser==1.1.4 >> >>>>> 14:19:22 pylint runtests: PYTHONHASHSEED='528910716' >> >>>>> 14:19:22 pylint runtests: commands[0] | pylint --errors-only >> static/usr/share/vdsm/sitecustomize.py lib/vdsm lib/vdsmclient lib/yajsonrpc >> >>>>> 14:19:23 Problem importing module base.pyc: cannot import name >> get_node_last_lineno >> >>>>> 14:19:24 Problem importing module base.py: cannot import name >> get_node_last_lineno >> >>>>> 14:19:34 06-27 14:18:09,674 DEBUG (MainThread) [Executor] Waiting >> for worker jsonrpc/7 (executor:290) >> >>>>> 14:19:34 2018-06-27 14:18:09,675 DEBUG (MainThread) [Executor] >> Waiting for worker jsonrpc/5 (executor:290) >> >>>>> 14:19:34 2018-06-27 14:18:09,675 DEBUG (MainThread) [Executor] >> Waiting for worker jsonrpc/2 (executor:290) >> >>>>> 14:19:34 2018-06-27 14:18:09,676 DEBUG (MainThread) [Executor] >> Waiting for worker jsonrpc/3 (executor:290) >> >>>>> 14:19:34 2018-06-27 14:18:09,676 DEBUG (MainThread) [Executor] >> Waiting for worker jsonrpc/4 (executor:290) >> >>>>> 14:19:34 2018-06-27 14:18:09,677 DEBUG (MainThread) [Scheduler] >> Stopping scheduler test.Scheduler (schedule:110) >> >>>>> 14:19:34 2018-06-27 14:18:09,678 DEBUG (test.Scheduler) >> [Scheduler] stopped (schedule:143) >> >>>>> 14:19:34 2018-06-27 14:18:09,679 DEBUG (test.Scheduler) >> [Scheduler] FINISH thread <Thread(test.Scheduler, started daemon >> 140562082535168)> (concurrent:196) >> >>>>> 14:19:34 --------------------- >> end captured logging << >> --------------------- >> >>>>> >> >>>>> >> > >> > _______________________________________________ >> > Devel mailing list -- devel(a)ovirt.org >> > To unsubscribe send an email to devel-leave(a)ovirt.org >> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> > oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> > List Archives: >> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ACPGZ4N3ULQ... >> _______________________________________________ >> Devel mailing list -- devel(a)ovirt.org >> To unsubscribe send an email to devel-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/IJJZC7GEB2J... > > > Sorry to bring this back, I see Piotr patch was not merged and the > problem still happens. Just hit it on latest master on fc28: > > https://jenkins.ovirt.org/job/vdsm_master_check-patch-fc28-x86_64/1808/ar... > > When submitting longer series of patches, these random errors are really > annoying as 1 or 2 patches always fail. Then submit again and some other > patch fails... > I rebased the patch. @Nir, @Milan, @Marcin Please review

On Wed, Nov 7, 2018 at 1:16 PM Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > On Wed, Nov 7, 2018 at 10:13 AM Piotr Kliczewski <pkliczew(a)redhat.com> wrote: >>>> On Wed, Nov 7, 2018 at 5:21 AM Germano Veit Michel <germano(a)redhat.com>> wrote: >>>>>> On Sat, Jun 30, 2018 at 12:43 AM Piotr Kliczewski < >>> piotr.kliczewski(a)gmail.com&gt; wrote: >>>>>> Here is the patch [1] which should fix it. >>>>>>> [1] https://gerrit.ovirt.org/#/c/92690/ >>>> On Thu, Jun 28, 2018 at 2:44 PM Piotr Kliczewski <pkliczew(a)redhat.com>>>> wrote: >>>> >>>> > Nir, >>>> >>>> > It looks like we have a race condition where request arrive sooner >>>> than subscribe frame: >>>> >>>> > 16:19:02 2018-06-27 14:17:54,653 INFO (jsonrpc/0) >>>> [jsonrpc.JsonRpcServer] RPC call echo succeeded in 0.01 seconds >>>> (__init__:311) >>>> > 16:19:02 2018-06-27 14:17:54,661 INFO (Detector thread) >>>> [Broker.StompAdapter] Subscribe command received (stompserver:123) >>>> >>>> > Thanks for reporting I will push a patch to fix it. >>>> >>>> > Thanks, >>>> > Piotr >>>> >>>> >>>> > On Wed, Jun 27, 2018 at 5:40 PM, Piotr Kliczewski < >>>> pkliczew(a)redhat.com&gt; wrote: >>>> >>>>> >> Ok, I'll take a look. >>>> >>>>> >> śr., 27 cze 2018, 17:36 użytkownik Nir Soffer <nsoffer(a)redhat.com>>>> napisał: >>>> >>>>>> >>> On Wed, Jun 27, 2018 at 6:13 PM Piotr Kliczewski < >>>> pkliczew(a)redhat.com&gt; wrote: >>>> >>>>>>> >>>> On Wed, Jun 27, 2018 at 5:01 PM, Nir Soffer <nsoffer(a)redhat.com>>>> wrote: >>>> >>>>>>>> >>>>> This tests used to fail in the past, but since we fixed it or >>>> the code >>>> >>>>>>>> >>>>> it never failed. >>>> >>>>>>>> >>>>>>>> >>>>> Maybe the slave was overloaded? >>>> >>>>>>> >>>>>>> >>>> Very possible. Can you paste a link to the job which failed? >>>> >>>>>> >>>>>> >>> Here: >>>> https://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/24130/ >>>> >>>>>> >>> The next build passed. >>>> >>>>>> >>> Maybe we need to solve the flakiness of some tests by running a >>>> flaky test >>>> >>> again, and let the build fail only if a test failed twice. I >>>> wonder if there is some >>>> >>> pytest plugin doing this. >>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>> 14:19:02 >>>> ====================================================================== >>>> >>>>> 14:19:02 ERROR: >>>> >>>>>>>> >>>>> 14:19:02 >>>> ---------------------------------------------------------------------- >>>> >>>>> 14:19:02 Traceback (most recent call last): >>>> >>>>> 14:19:02 File >>>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/testlib.py", >>>> line 143, in wrapper >>>> >>>>> 14:19:02 return f(self, *args) >>>> >>>>> 14:19:02 File >>>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/stomp_test.py", >>>> line 95, in test_echo >>>> >>>>> 14:19:02 str(uuid4())), >>>> >>>>> 14:19:02 File >>>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/lib/yajsonrpc/jsonrpcclient.py", >>>> line 77, in callMethod >>>> >>>>> 14:19:02 raise >>>> exception.JsonRpcNoResponseError(method=methodName) >>>> >>>>> 14:19:02 JsonRpcNoResponseError: No response for JSON-RPC >>>> request: {'method': 'echo'} >>>> >>>>> 14:19:02 -------------------- >> begin captured logging << >>>> -------------------- >>>> >>>>> 14:19:02 2018-06-27 14:17:54,524 DEBUG (MainThread) >>>> [vds.MultiProtocolAcceptor] Creating socket (host='::1', port=0, family=10, >>>> socketype=1, proto=6) (protocoldetector:225) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,526 INFO (MainThread) >>>> [vds.MultiProtocolAcceptor] Listening at ::1:36713 (protocoldetector:183) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,535 DEBUG (MainThread) [Scheduler] >>>> Starting scheduler test.Scheduler (schedule:98) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,537 DEBUG (test.Scheduler) >>>> [Scheduler] START thread <Thread(test.Scheduler, started daemon >>>> 140562082535168)> (func=<bound method Scheduler._run of >>>> <vdsm.schedule.Scheduler object at 0x7fd74ca00390>>, args=(), kwargs={}) >>>> (concurrent:193) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,538 DEBUG (test.Scheduler) >>>> [Scheduler] started (schedule:140) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,546 DEBUG (JsonRpc (StompReactor)) >>>> [root] START thread <Thread(JsonRpc (StompReactor), started daemon >>>> 140562629256960)> (func=<bound method StompReactor.process_requests of >>>> <yajsonrpc.stompserver.StompReactor object at 0x7fd74ca006d0>>, args=(), >>>> kwargs={}) (concurrent:193) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,547 DEBUG (MainThread) [Executor] >>>> Starting executor (executor:128) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,549 DEBUG (MainThread) [Executor] >>>> Starting worker jsonrpc/0 (executor:286) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,553 DEBUG (jsonrpc/0) [Executor] >>>> START thread <Thread(jsonrpc/0, started daemon 140562612471552)>>>> (func=<bound method _Worker._run of <Worker name=jsonrpc/0 waiting task#=0 >>>> at 0x7fd74ca00650>>, args=(), kwargs={}) (concurrent:193) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,554 DEBUG (jsonrpc/0) [Executor] >>>> Worker started (executor:298) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,557 DEBUG (MainThread) [Executor] >>>> Starting worker jsonrpc/1 (executor:286) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,558 DEBUG (MainThread) [Executor] >>>> Starting worker jsonrpc/2 (executor:286) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,559 DEBUG (jsonrpc/2) [Executor] >>>> START thread <Thread(jsonrpc/2, started daemon 140562620864256)>>>> (func=<bound method _Worker._run of <Worker name=jsonrpc/2 waiting task#=0 >>>> at 0x7fd74c9fb350>>, args=(), kwargs={}) (concurrent:193) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,560 DEBUG (jsonrpc/2) [Executor] >>>> Worker started (executor:298) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,561 DEBUG (MainThread) [Executor] >>>> Starting worker jsonrpc/3 (executor:286) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,562 DEBUG (jsonrpc/3) [Executor] >>>> START thread <Thread(jsonrpc/3, started daemon 140562124498688)>>>> (func=<bound method _Worker._run of <Worker name=jsonrpc/3 waiting task#=0 >>>> at 0x7fd74bc80290>>, args=(), kwargs={}) (concurrent:193) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,563 DEBUG (jsonrpc/3) [Executor] >>>> Worker started (executor:298) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,564 DEBUG (MainThread) [Executor] >>>> Starting worker jsonrpc/4 (executor:286) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,565 DEBUG (jsonrpc/4) [Executor] >>>> START thread <Thread(jsonrpc/4, started daemon 140562116105984)>>>> (func=<bound method _Worker._run of <Worker name=jsonrpc/4 waiting task#=0 >>>> at 0x7fd74bc80790>>, args=(), kwargs={}) (concurrent:193) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,566 DEBUG (jsonrpc/4) [Executor] >>>> Worker started (executor:298) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,568 DEBUG (jsonrpc/1) [Executor] >>>> START thread <Thread(jsonrpc/1, started daemon 140562132891392)>>>> (func=<bound method _Worker._run of <Worker name=jsonrpc/1 waiting task#=0 >>>> at 0x7fd74c9fb690>>, args=(), kwargs={}) (concurrent:193) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,569 DEBUG (jsonrpc/1) [Executor] >>>> Worker started (executor:298) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,569 DEBUG (MainThread) [Executor] >>>> Starting worker jsonrpc/5 (executor:286) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,570 DEBUG (jsonrpc/5) [Executor] >>>> START thread <Thread(jsonrpc/5, started daemon 140562107713280)>>>> (func=<bound method _Worker._run of <Worker name=jsonrpc/5 waiting task#=0 >>>> at 0x7fd74bc80090>>, args=(), kwargs={}) (concurrent:193) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,571 DEBUG (jsonrpc/5) [Executor] >>>> Worker started (executor:298) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,572 DEBUG (MainThread) [Executor] >>>> Starting worker jsonrpc/6 (executor:286) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,580 DEBUG (MainThread) [Executor] >>>> Starting worker jsonrpc/7 (executor:286) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,603 DEBUG (jsonrpc/6) [Executor] >>>> START thread <Thread(jsonrpc/6, started daemon 140562099320576)>>>> (func=<bound method _Worker._run of <Worker name=jsonrpc/6 waiting task#=0 >>>> at 0x7fd74d0dacd0>>, args=(), kwargs={}) (concurrent:193) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,604 DEBUG (jsonrpc/6) [Executor] >>>> Worker started (executor:298) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,615 DEBUG (jsonrpc/7) [Executor] >>>> START thread <Thread(jsonrpc/7, started daemon 140562090927872)>>>> (func=<bound method _Worker._run of <Worker name=jsonrpc/7 waiting task#=0 >>>> at 0x7fd74bfe0310>>, args=(), kwargs={}) (concurrent:193) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,616 DEBUG (JsonRpcServer) [root] >>>> START thread <Thread(JsonRpcServer, started daemon 140561730238208)>>>> (func=<bound method JsonRpcServer.serve_requests of >>>> <yajsonrpc.JsonRpcServer object at 0x7fd74ca00690>>, args=(), kwargs={}) >>>> (concurrent:193) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,618 DEBUG (MainThread) >>>> [vds.MultiProtocolAcceptor] Adding detector >>>> <yajsonrpc.stompserver.StompDetector instance at 0x7fd74c28cc68>>>> (protocoldetector:210) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,625 INFO (Detector thread) >>>> [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:52432 >>>> (protocoldetector:61) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,626 DEBUG (Detector thread) >>>> [ProtocolDetector.Detector] Using required_size=11 (protocoldetector:89) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,627 DEBUG (jsonrpc/7) [Executor] >>>> Worker started (executor:298) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,634 DEBUG (MainThread) >>>> [jsonrpc.AsyncoreClient] Sending response (stompclient:293) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,634 DEBUG (Client ::1:36713) [root] >>>> START thread <Thread(Client ::1:36713, started daemon 140561713452800)>>>> (func=<bound method Reactor.process_requests of >>>> <yajsonrpc.betterAsyncore.Reactor object at 0x7fd74bdb1d90>>, args=(), >>>> kwargs={}) (concurrent:193) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,641 INFO (Detector thread) >>>> [ProtocolDetector.Detector] Detected protocol stomp from ::1:52432 >>>> (protocoldetector:125) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,645 INFO (Detector thread) >>>> [Broker.StompAdapter] Processing CONNECT request (stompserver:94) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,648 DEBUG (Client ::1:36713) >>>> [yajsonrpc.protocols.stomp.AsyncClient] Stomp connection established >>>> (stompclient:137) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,651 DEBUG (jsonrpc/0) >>>> [jsonrpc.JsonRpcServer] Calling 'echo' in bridge with [u'Lorem ipsum dolor >>>> sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut >>>> labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud >>>> exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis >>>> aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu >>>> fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt >>>> in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor >>>> sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut >>>> labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud >>>> exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis >>>> aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu >>>> fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt >>>> in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor >>>> sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut >>>> labore et dolore magna aliqua. Ut en'] (__init__:328) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,652 DEBUG (jsonrpc/0) >>>> [jsonrpc.JsonRpcServer] Return 'echo' in bridge with Lorem ipsum dolor sit >>>> amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut >>>> labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud >>>> exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis >>>> aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu >>>> fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt >>>> in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor >>>> sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut >>>> labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud >>>> exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis >>>> aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu >>>> fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt >>>> in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor >>>> sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut >>>> labore et dolore magna aliqua. Ut en (__init__:355) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,653 INFO (jsonrpc/0) >>>> [jsonrpc.JsonRpcServer] RPC call echo succeeded in 0.01 seconds >>>> (__init__:311) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,661 INFO (Detector thread) >>>> [Broker.StompAdapter] Subscribe command received (stompserver:123) >>>> >>>>> 14:19:02 2018-06-27 14:17:54,662 DEBUG (Detector thread) >>>> [protocoldetector.StompDetector] Stomp detected from ('::1', 52432) >>>> (stompserver:412) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,649 DEBUG (MainThread) >>>> [vds.MultiProtocolAcceptor] Stopping Acceptor (protocoldetector:214) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,650 INFO (MainThread) >>>> [jsonrpc.JsonRpcServer] Stopping JsonRPC Server (__init__:441) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,651 DEBUG (MainThread) [Executor] >>>> Stopping executor (executor:137) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,652 DEBUG (Client ::1:36713) [root] >>>> FINISH thread <Thread(Client ::1:36713, started daemon 140561713452800)>>>> (concurrent:196) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,652 DEBUG (JsonRpcServer) [root] >>>> FINISH thread <Thread(JsonRpcServer, started daemon 140561730238208)>>>> (concurrent:196) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,654 DEBUG (JsonRpc (StompReactor)) >>>> [root] FINISH thread <Thread(JsonRpc (StompReactor), started daemon >>>> 140562629256960)> (concurrent:196) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,655 DEBUG (jsonrpc/2) [Executor] >>>> Worker stopped (executor:303) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,658 DEBUG (jsonrpc/3) [Executor] >>>> Worker stopped (executor:303) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,659 DEBUG (jsonrpc/4) [Executor] >>>> Worker stopped (executor:303) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,660 DEBUG (jsonrpc/1) [Executor] >>>> Worker stopped (executor:303) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,660 DEBUG (jsonrpc/5) [Executor] >>>> Worker stopped (executor:303) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,660 DEBUG (jsonrpc/6) [Executor] >>>> Worker stopped (executor:303) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,665 DEBUG (jsonrpc/6) [Executor] >>>> FINISH thread <Thread(jsonrpc/6, started daemon 140562099320576)>>>> (concurrent:196) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,661 DEBUG (jsonrpc/7) [Executor] >>>> Worker stopped (executor:303) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,662 DEBUG (jsonrpc/2) [Executor] >>>> FINISH thread <Thread(jsonrpc/2, started daemon 140562620864256)>>>> (concurrent:196) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,663 DEBUG (jsonrpc/3) [Executor] >>>> FINISH thread <Thread(jsonrpc/3, started daemon 140562124498688)>>>> (concurrent:196) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,663 DEBUG (jsonrpc/1) [Executor] >>>> FINISH thread <Thread(jsonrpc/1, started daemon 140562132891392)>>>> (concurrent:196) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,664 DEBUG (jsonrpc/4) [Executor] >>>> FINISH thread <Thread(jsonrpc/4, started daemon 140562116105984)>>>> (concurrent:196) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,664 DEBUG (jsonrpc/0) [Executor] >>>> Worker stopped (executor:303) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,665 DEBUG (jsonrpc/5) [Executor] >>>> FINISH thread <Thread(jsonrpc/5, started daemon 140562107713280)>>>> (concurrent:196) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,667 DEBUG (jsonrpc/7) [Executor] >>>> FINISH thread <Thread(jsonrpc/7, started daemon 140562090927872)>>>> (concurrent:196) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,667 DEBUG (MainThread) [Executor] >>>> Waiting for worker jsonrpc/0 (executor:290) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,671 DEBUG (jsonrpc/0) [Executor] >>>> FINISH thread <Thread(jsonrpc/0, started daemon 140562612471552)>>>> (concurrent:196) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,673 DEBUG (MainThread) [Executor] >>>> Waiting for worker jsonrpc/1 (executor:290) >>>> >>>>> 14:19:02 2018-06-27 14:18:09,674 DEBUG (MainThread) [Executor] >>>> Waiting for worker jsonrpc/6 (executor:290) >>>> >>>>> 14:19:02 2018-Coverage.py warning: Module >>>> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm was >>>> never imported. (module-not-imported) >>>> >>>>> 14:19:05 pylint installdeps: pylint==1.8 >>>> >>>>> 14:19:22 pylint installed: The directory >>>> '/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/.cache/pip/http' >>>> or its parent directory is not owned by the current user and the cache has >>>> been disabled. Please check the permissions and owner of that directory. If >>>> executing pip with sudo, you may want sudo's -H >>>> flag.,astroid==1.6.5,backports.functools-lru-cache==1.5,backports.ssl-match-hostname==3.5.0.1,blivet==0.61.15.69,chardet==2.2.1,configparser==3.5.0,coverage==4.4.1,decorator==3.4.0,enum34==1.1.6,funcsigs==1.0.2,futures==3.2.0,iniparse==0.4,ioprocess==1.1.0,ipaddress==1.0.16,IPy==0.75,isort==4.3.4,kitchen==1.1.1,lazy-object-proxy==1.3.1,libvirt-python==3.9.0,Magic-file-extensions==0.2,mccabe==0.6.1,mock==2.0.0,netaddr==0.7.18,ovirt-imageio-common==1.4.1,pbr==3.1.1,pluggy==0.6.0,policycoreutils-default-encoding==0.1,pthreading==0.1.3,py==1.5.4,pycurl==7.19.0,pygpgme==0.3,pyinotify==0.9.4,pykickstart==1.99.66.18,pyliblzma==0.5.3,pylint==1.8.0,pyparted==3.9,python-augeas==0.5.0,python-dateutil==2.4.2,pyudev==0.15,pyxattr==0.5.1,PyYAML==3.10,requests==2.6.0,sanlock-python==3.6.0,seobject==0.1,sepolicy==1.1,singledispatch==3.4.0.3,six==1.11.0,subprocess32==3.2.6,tox==2.9.1,urlgrabber==3.10,urllib3==1.10.2,virtualenv==16.0.0,WebOb==1.2.3,wrapt==1.10.11,yum-metadata-parser==1.1.4 >>>> >>>>> 14:19:22 pylint runtests: PYTHONHASHSEED='528910716' >>>> >>>>> 14:19:22 pylint runtests: commands[0] | pylint --errors-only >>>> static/usr/share/vdsm/sitecustomize.py lib/vdsm lib/vdsmclient lib/yajsonrpc >>>> >>>>> 14:19:23 Problem importing module base.pyc: cannot import name >>>> get_node_last_lineno >>>> >>>>> 14:19:24 Problem importing module base.py: cannot import name >>>> get_node_last_lineno >>>> >>>>> 14:19:34 06-27 14:18:09,674 DEBUG (MainThread) [Executor] >>>> Waiting for worker jsonrpc/7 (executor:290) >>>> >>>>> 14:19:34 2018-06-27 14:18:09,675 DEBUG (MainThread) [Executor] >>>> Waiting for worker jsonrpc/5 (executor:290) >>>> >>>>> 14:19:34 2018-06-27 14:18:09,675 DEBUG (MainThread) [Executor] >>>> Waiting for worker jsonrpc/2 (executor:290) >>>> >>>>> 14:19:34 2018-06-27 14:18:09,676 DEBUG (MainThread) [Executor] >>>> Waiting for worker jsonrpc/3 (executor:290) >>>> >>>>> 14:19:34 2018-06-27 14:18:09,676 DEBUG (MainThread) [Executor] >>>> Waiting for worker jsonrpc/4 (executor:290) >>>> >>>>> 14:19:34 2018-06-27 14:18:09,677 DEBUG (MainThread) [Scheduler] >>>> Stopping scheduler test.Scheduler (schedule:110) >>>> >>>>> 14:19:34 2018-06-27 14:18:09,678 DEBUG (test.Scheduler) >>>> [Scheduler] stopped (schedule:143) >>>> >>>>> 14:19:34 2018-06-27 14:18:09,679 DEBUG (test.Scheduler) >>>> [Scheduler] FINISH thread <Thread(test.Scheduler, started daemon >>>> 140562082535168)> (concurrent:196) >>>> >>>>> 14:19:34 --------------------- >> end captured logging << >>>> --------------------- >>>> >>>>>>>> >>>>>>>> >>>> > _______________________________________________ >>>> > Devel mailing list -- devel(a)ovirt.org >>>> > To unsubscribe send an email to devel-leave(a)ovirt.org >>>> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> > oVirt Code of Conduct: >>>> https://www.ovirt.org/community/about/community-guidelines/ >>>> > List Archives: >>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ACPGZ4N3ULQ... >>>> _______________________________________________ >>>> Devel mailing list -- devel(a)ovirt.org >>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> oVirt Code of Conduct: >>>> https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: >>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/IJJZC7GEB2J... >>>>>>> Sorry to bring this back, I see Piotr patch was not merged and the >>> problem still happens. Just hit it on latest master on fc28: >>>>> https://jenkins.ovirt.org/job/vdsm_master_check-patch-fc28-x86_64/1808/ar... >>>>> When submitting longer series of patches, these random errors are >>> really annoying as 1 or 2 patches always fail. Then submit again and some >>> other patch fails... >>>>> I rebased the patch. @Nir, @Milan, @Marcin Please review >> I don't think this is the right approach. This does not fix the root cause > but cover it by playing with order of messages. Please read the thread again and you will see why the order matters here.