On Mon, Aug 14, 2017 at 4:26 PM pengyixiang <yxpengi386@163.com> wrote:

hello everyone!
Now my main way to learn and debug vdsm is to write pdb code to vdsm and restart vdsmd.service, dose it have a better way? As some ide that is more convenient to add breaking point、step into, etc.

The best way to debug vdsm is to write good tests. When you fail to write tests

or in untested areas, your best way is to use logs.

Stopping vdsm in the debugger may cause engine to fence it, and in general

stopping entire multi threaded program like vdsm is not very useful.

Another way to debug is via manhole - you can get a python shell inside vdsm,

and poke in its internals.

To use manhole install it:

# pip install -U manhole

And enable using vdsm dropin conf - create a file:

# cat /etc/vdsm/vdsm.conf.d/manhole.conf

[devel]

# Enable manhole debugging service (requires manhole package).

manhole_enable = true

And restart vdsm.

To open a python shell inside vdsm you can use netcat, but socat is

much better because of the readline integration.

# socat readline unix-connect:/run/vdsm/vdsmd.manhole

######### ProcessID=21004, ThreadID=139830629488384 #########

File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap

self.__bootstrap_inner()

File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner

self.run()

File: "/usr/lib/python2.7/site-packages/manhole.py", line 225, in run

self.handle(self.client, self.locals)

File: "/usr/lib/python2.7/site-packages/manhole.py", line 268, in handle

run_repl(locals)

File: "/usr/lib/python2.7/site-packages/manhole.py", line 313, in run_repl

dump_stacktraces()

File: "/usr/lib/python2.7/site-packages/manhole.py", line 578, in dump_stacktraces

for filename, lineno, name, line in traceback.extract_stack(stack):

[--snipped stacktrace of all threads--]

Python 2.7.5 (default, May 3 2017, 07:55:04)

[GCC 4.8.5 20150623 (Red Hat 4.8.5-14)] on linux2

Type "help", "copyright", "credits" or "license" for more information.

(ManholeConsole)

>>>

Checking your local namespace:

>>> dir()

['__builtins__', 'cif', 'dump_stacktraces', 'irs', 'os', 'socket', 'sys', 'traceback']

irs is the storage endpoint, you can look into it:

>>> irs._obj.domainMonitor._monitors

{u'b9cda581-f9ac-4650-83e0-17d00d7dfc75': <vdsm.storage.monitor.MonitorThread object at 0x3ecd6d0>, u'd4ad5a78-1d75-457b-8caf-400d2d7917a6': <vdsm.storage.monitor.MonitorThread object at 0x3e38890>, u'd6e4a622-bd31-4d8f-904d-1e26b7286757': <vdsm.storage.monitor.MonitorThread object at 0x3f2bf50>, u'6ffbc483-0031-403a-819b-3bb2f0f8de0a': <vdsm.storage.monitor.MonitorThread object at 0x3f2b5d0>, u'373e8c55-283f-41d4-8433-95c1ef1bbd1a': <vdsm.storage.monitor.MonitorThread object at 0x3ecde10>, u'2770b49f-79d7-4b7c-9d39-03e988e8bc4d': <vdsm.storage.monitor.MonitorThread object at 0x3f37690>}

>>> irs._obj.domainMonitor._monitors['d6e4a622-bd31-4d8f-904d-1e26b7286757'].status.readDelay

0.000905336

>>> irs._obj.domainMonitor._checker._loop._scheduled

[<Timer when=4920799.730000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2af6110> at 0x2396650>, <Timer when=4920799.750000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2afc890> at 0x2396d50>, <Timer when=4920800.690000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2b28890> at 0x2396590>, <Timer when=4920799.770000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2b21c90> at 0x2331a90>, <Timer when=4920800.620000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2b21d90> at 0x2a997d0>, <Timer when=4920800.800000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2b9a9d0> at 0x23967d0>]

>>> irs._obj.domainMonitor._checker._loop._scheduled[0]._callback._callback

dir(obj) is your best friend inside the manhole.

Other ways to debug vdsm:

- ps auxf - see vdsm child processes

- htop - you can enable thread names and use tree mode to see which vdsm thread is consuming lot of cpu

- strace - you can see which system calls vdsm is invoking

- gdb - you can attach to vdsm and get a python backtrace using py-bt

Nir

Have a nice day
--------------------------------
pencc
_______________________________________________
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel