[Users] Fwd: Re: latest vdsm cannot read ib device speeds causing storage attach fail

Hi, I reproduced this issue, and I believe it's a python bug. 1. How to reproduce: with the test case attached, put it under /usr/share/vdsm/tests/, run #./run_tests.sh superVdsmTests.py and this issue will be reproduced. 2.Log analyse: We notice a strange pattern in this log: connectStorageServer be called twice, first supervdsm call succeed, second fails becasue of validateAccess(). That is because for the first call validateAccess returns normally and leave a child there, when the second validateAccess call arrives and multirprocessing manager is receiving the method message, it is just the time first child exit and SIGCHLD comming, this signal interrupted multiprocessing receive system call, python managers.py should handle INTR and retry recv() like we do in vdsm but it's not, so the second one raise error. the vdsm side receive RemoteError because of supervdsm server multiprocessing manager raise error KIND='TRACEBACK' The upper part is the trace back from the client side, the following part is from server side: Corresponding Python source code:managers.py(Server side) def serve_client(self, conn): ''' Handle requests from the proxies in a particular process/thread ''' util.debug('starting server thread to service %r', threading.current_thread().name) recv = conn.recv send = conn.send id_to_obj = self.id_to_obj while not self.stop: try: methodname = obj = None request = recv()<------------------this line been interrupted by SIGCHLD ident, methodname, args, kwds = request obj, exposed, gettypeid = id_to_obj[ident] if methodname not in exposed: raise AttributeError( 'method %r of %r object is not in exposed=%r' % (methodname, type(obj), exposed) ) function = getattr(obj, methodname) try: res = function(*args, **kwds) except Exception, e: msg = ('#ERROR', e) else: typeid = gettypeid and gettypeid.get(methodname, None) if typeid: rident, rexposed = self.create(conn, typeid, res) token = Token(typeid, self.address, rident) msg = ('#PROXY', (rexposed, token)) else: msg = ('#RETURN', res) except AttributeError: if methodname is None: msg = ('#TRACEBACK', format_exc()) else: try: fallback_func = self.fallback_mapping[methodname] result = fallback_func( self, conn, ident, obj, *args, **kwds ) msg = ('#RETURN', result) except Exception: msg = ('#TRACEBACK', format_exc()) except EOFError: util.debug('got EOF -- exiting thread serving %r', threading.current_thread().name) sys.exit(0) except Exception:<------does not handle IOError,INTR here should retry recv() msg = ('#TRACEBACK', format_exc()) 3. Actions we will take: (1)As a work round we can first remove the zombie reaper from supervdsm server (2)I'll see whether python has a fixed version for this (3)Yaniv is working on changing vdsm/svdsm communication channel to pipe and handle it ourselves, I believe we'll get rid of this with that properly handled. -------- Original Message -------- Subject: Re: [Users] latest vdsm cannot read ib device speeds causing storage attach fail Resent-Date: Thu, 24 Jan 2013 12:24:10 +0200 Resent-From: Dan Kenigsberg <danken(a)redhat.com&gt; Resent-To: Royce Lv <lvroyce(a)linux.vnet.ibm.com&gt; Date: Wed, 23 Jan 2013 10:44:57 -0600 From: Dead Horse <deadhorseconsulting(a)gmail.com&gt; To: Dan Kenigsberg <danken(a)redhat.com&gt; CC: <users(a)ovirt.org&gt; <users(a)ovirt.org&gt; VDSM was built from: commit 166138e37e75767b32227746bb671b1dab9cdd5e Attached is the full vdsm log I should also note that from engine perspective it sees the master storage domain as locked and the others as unknown. On Wed, Jan 23, 2013 at 2:49 AM, Dan Kenigsberg <danken(a)redhat.com&gt; wrote: