On Mon, Aug 14, 2017 at 4:26 PM pengyixiang <yxpengi386@163.com> wrote:
hello everyone!

       Now my main way to learn and debug vdsm is to write pdb code to vdsm and restart vdsmd.service, dose it have a better way? As some ide that is more convenient to add breaking point、step into, etc. 

The best way to debug vdsm is to write good tests. When you fail to write tests
or in untested areas, your best way is to use logs.

Stopping vdsm in the debugger may cause engine to fence it, and in general 
stopping entire multi threaded program like vdsm is not very useful.

Another way to debug is via manhole - you can get a python shell inside vdsm,
and poke in its internals.

To use manhole install it:

# pip install -U manhole

And enable using vdsm dropin conf - create a file:

# cat /etc/vdsm/vdsm.conf.d/manhole.conf 
# Enable manhole debugging service (requires manhole package).
manhole_enable = true

And restart vdsm.

To open a python shell inside vdsm you can use netcat, but socat is
much better because of the readline integration.

# socat readline unix-connect:/run/vdsm/vdsmd.manhole

######### ProcessID=21004, ThreadID=139830629488384 #########
File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap
File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
File: "/usr/lib/python2.7/site-packages/manhole.py", line 225, in run
  self.handle(self.client, self.locals)
File: "/usr/lib/python2.7/site-packages/manhole.py", line 268, in handle
File: "/usr/lib/python2.7/site-packages/manhole.py", line 313, in run_repl
File: "/usr/lib/python2.7/site-packages/manhole.py", line 578, in dump_stacktraces
  for filename, lineno, name, line in traceback.extract_stack(stack):

[--snipped stacktrace of all threads--]

Python 2.7.5 (default, May  3 2017, 07:55:04) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-14)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Checking your local namespace:

>>> dir()
['__builtins__', 'cif', 'dump_stacktraces', 'irs', 'os', 'socket', 'sys', 'traceback']

irs is the storage endpoint, you can look into it:

>>> irs._obj.domainMonitor._monitors
{u'b9cda581-f9ac-4650-83e0-17d00d7dfc75': <vdsm.storage.monitor.MonitorThread object at 0x3ecd6d0>, u'd4ad5a78-1d75-457b-8caf-400d2d7917a6': <vdsm.storage.monitor.MonitorThread object at 0x3e38890>, u'd6e4a622-bd31-4d8f-904d-1e26b7286757': <vdsm.storage.monitor.MonitorThread object at 0x3f2bf50>, u'6ffbc483-0031-403a-819b-3bb2f0f8de0a': <vdsm.storage.monitor.MonitorThread object at 0x3f2b5d0>, u'373e8c55-283f-41d4-8433-95c1ef1bbd1a': <vdsm.storage.monitor.MonitorThread object at 0x3ecde10>, u'2770b49f-79d7-4b7c-9d39-03e988e8bc4d': <vdsm.storage.monitor.MonitorThread object at 0x3f37690>}

>>> irs._obj.domainMonitor._monitors['d6e4a622-bd31-4d8f-904d-1e26b7286757'].status.readDelay

>>> irs._obj.domainMonitor._checker._loop._scheduled
[<Timer when=4920799.730000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2af6110> at 0x2396650>, <Timer when=4920799.750000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2afc890> at 0x2396d50>, <Timer when=4920800.690000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2b28890> at 0x2396590>, <Timer when=4920799.770000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2b21c90> at 0x2331a90>, <Timer when=4920800.620000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2b21d90> at 0x2a997d0>, <Timer when=4920800.800000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2b9a9d0> at 0x23967d0>]

>>> irs._obj.domainMonitor._checker._loop._scheduled[0]._callback._callback
<bound method DirectioChecker._check of <DirectioChecker /rhev/data-center/mnt/dumbo.tlv.redhat.com:_voodoo_41/2770b49f-79d7-4b7c-9d39-03e988e8bc4d/dom_md/metadata running next_check=4920929.73 at 0x2af6050>>

dir(obj) is your best friend inside the manhole.

Other ways to debug vdsm:
- ps auxf - see vdsm child processes
- htop - you can enable thread names and use tree mode to see which vdsm thread is consuming lot of cpu
- strace  - you can see which system calls vdsm is invoking
- gdb - you can attach to vdsm and get a python backtrace using py-bt


Have a nice day


Devel mailing list