On Mon, Aug 14, 2017 at 4:26 PM pengyixiang <yxpengi386(a)163.com&gt; wrote: The best way to debug vdsm is to write good tests. When you fail to write tests or in untested areas, your best way is to use logs. Stopping vdsm in the debugger may cause engine to fence it, and in general stopping entire multi threaded program like vdsm is not very useful. Another way to debug is via manhole - you can get a python shell inside vdsm, and poke in its internals. To use manhole install it: # pip install -U manhole And enable using vdsm dropin conf - create a file: # cat /etc/vdsm/vdsm.conf.d/manhole.conf [devel] # Enable manhole debugging service (requires manhole package). manhole_enable = true And restart vdsm. To open a python shell inside vdsm you can use netcat, but socat is much better because of the readline integration. # socat readline unix-connect:/run/vdsm/vdsmd.manhole ######### ProcessID=21004, ThreadID=139830629488384 ######### File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap self.__bootstrap_inner() File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner self.run() File: "/usr/lib/python2.7/site-packages/manhole.py", line 225, in run self.handle(self.client, self.locals) File: "/usr/lib/python2.7/site-packages/manhole.py", line 268, in handle run_repl(locals) File: "/usr/lib/python2.7/site-packages/manhole.py", line 313, in run_repl dump_stacktraces() File: "/usr/lib/python2.7/site-packages/manhole.py", line 578, in dump_stacktraces for filename, lineno, name, line in traceback.extract_stack(stack): [--snipped stacktrace of all threads--] Python 2.7.5 (default, May 3 2017, 07:55:04) [GCC 4.8.5 20150623 (Red Hat 4.8.5-14)] on linux2 Type "help", "copyright", "credits" or "license" for more information. (ManholeConsole) Checking your local namespace: ['__builtins__', 'cif', 'dump_stacktraces', 'irs', 'os', 'socket', 'sys', 'traceback'] irs is the storage endpoint, you can look into it: {u'b9cda581-f9ac-4650-83e0-17d00d7dfc75': <vdsm.storage.monitor.MonitorThread object at 0x3ecd6d0>, u'd4ad5a78-1d75-457b-8caf-400d2d7917a6': <vdsm.storage.monitor.MonitorThread object at 0x3e38890>, u'd6e4a622-bd31-4d8f-904d-1e26b7286757': <vdsm.storage.monitor.MonitorThread object at 0x3f2bf50>, u'6ffbc483-0031-403a-819b-3bb2f0f8de0a': <vdsm.storage.monitor.MonitorThread object at 0x3f2b5d0>, u'373e8c55-283f-41d4-8433-95c1ef1bbd1a': <vdsm.storage.monitor.MonitorThread object at 0x3ecde10>, u'2770b49f-79d7-4b7c-9d39-03e988e8bc4d': <vdsm.storage.monitor.MonitorThread object at 0x3f37690>} irs._obj.domainMonitor._monitors['d6e4a622-bd31-4d8f-904d-1e26b7286757'].status.readDelay 0.000905336 [<Timer when=4920799.730000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2af6110> at 0x2396650>, <Timer when=4920799.750000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2afc890> at 0x2396d50>, <Timer when=4920800.690000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2b28890> at 0x2396590>, <Timer when=4920799.770000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2b21c90> at 0x2331a90>, <Timer when=4920800.620000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2b21d90> at 0x2a997d0>, <Timer when=4920800.800000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2b9a9d0> at 0x23967d0>] <bound method DirectioChecker._check of <DirectioChecker /rhev/data-center/mnt/dumbo.tlv.redhat.com:_voodoo_41/2770b49f-79d7-4b7c-9d39-03e988e8bc4d/dom_md/metadata running next_check=4920929.73 at 0x2af6050>> dir(obj) is your best friend inside the manhole. Other ways to debug vdsm: - ps auxf - see vdsm child processes - htop - you can enable thread names and use tree mode to see which vdsm thread is consuming lot of cpu - strace - you can see which system calls vdsm is invoking - gdb - you can attach to vdsm and get a python backtrace using py-bt Nir