[ovirt-devel] ways to dev and debug vdsm

[Nir Soffer]

On Mon, Aug 14, 2017 at 4:26 PM pengyixiang <yxpengi386 at 163.com> wrote:

> hello everyone!
>
>        Now my main way to learn and debug vdsm is to write pdb code to
> vdsm and restart vdsmd.service, dose it have a better way? As some ide
> that is more convenient to add breaking point、step into, etc.
>

The best way to debug vdsm is to write good tests. When you fail to write
tests
or in untested areas, your best way is to use logs.

Stopping vdsm in the debugger may cause engine to fence it, and in general
stopping entire multi threaded program like vdsm is not very useful.

Another way to debug is via manhole - you can get a python shell inside
vdsm,
and poke in its internals.

To use manhole install it:

# pip install -U manhole

And enable using vdsm dropin conf - create a file:

# cat /etc/vdsm/vdsm.conf.d/manhole.conf
[devel]
# Enable manhole debugging service (requires manhole package).
manhole_enable = true

And restart vdsm.

To open a python shell inside vdsm you can use netcat, but socat is
much better because of the readline integration.

# socat readline unix-connect:/run/vdsm/vdsmd.manhole

######### ProcessID=21004, ThreadID=139830629488384 #########
File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap
  self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
  self.run()
File: "/usr/lib/python2.7/site-packages/manhole.py", line 225, in run
  self.handle(self.client, self.locals)
File: "/usr/lib/python2.7/site-packages/manhole.py", line 268, in handle
  run_repl(locals)
File: "/usr/lib/python2.7/site-packages/manhole.py", line 313, in run_repl
  dump_stacktraces()
File: "/usr/lib/python2.7/site-packages/manhole.py", line 578, in
dump_stacktraces
  for filename, lineno, name, line in traceback.extract_stack(stack):

[--snipped stacktrace of all threads--]

Python 2.7.5 (default, May  3 2017, 07:55:04)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-14)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(ManholeConsole)
>>>

Checking your local namespace:

>>> dir()
['__builtins__', 'cif', 'dump_stacktraces', 'irs', 'os', 'socket', 'sys',
'traceback']

irs is the storage endpoint, you can look into it:

>>> irs._obj.domainMonitor._monitors
{u'b9cda581-f9ac-4650-83e0-17d00d7dfc75':
<vdsm.storage.monitor.MonitorThread object at 0x3ecd6d0>,
u'd4ad5a78-1d75-457b-8caf-400d2d7917a6':
<vdsm.storage.monitor.MonitorThread object at 0x3e38890>,
u'd6e4a622-bd31-4d8f-904d-1e26b7286757':
<vdsm.storage.monitor.MonitorThread object at 0x3f2bf50>,
u'6ffbc483-0031-403a-819b-3bb2f0f8de0a':
<vdsm.storage.monitor.MonitorThread object at 0x3f2b5d0>,
u'373e8c55-283f-41d4-8433-95c1ef1bbd1a':
<vdsm.storage.monitor.MonitorThread object at 0x3ecde10>,
u'2770b49f-79d7-4b7c-9d39-03e988e8bc4d':
<vdsm.storage.monitor.MonitorThread object at 0x3f37690>}

>>>
irs._obj.domainMonitor._monitors['d6e4a622-bd31-4d8f-904d-1e26b7286757'].status.readDelay
0.000905336

>>> irs._obj.domainMonitor._checker._loop._scheduled
[<Timer when=4920799.730000 callback=<vdsm.storage.asyncutils.LoopingCall
object at 0x2af6110> at 0x2396650>, <Timer when=4920799.750000
callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2afc890> at
0x2396d50>, <Timer when=4920800.690000
callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2b28890> at
0x2396590>, <Timer when=4920799.770000
callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2b21c90> at
0x2331a90>, <Timer when=4920800.620000
callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2b21d90> at
0x2a997d0>, <Timer when=4920800.800000
callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2b9a9d0> at
0x23967d0>]

>>> irs._obj.domainMonitor._checker._loop._scheduled[0]._callback._callback
<bound method DirectioChecker._check of <DirectioChecker
/rhev/data-center/mnt/dumbo.tlv.redhat.com:_voodoo_41/2770b49f-79d7-4b7c-9d39-03e988e8bc4d/dom_md/metadata
running next_check=4920929.73 at 0x2af6050>>

dir(obj) is your best friend inside the manhole.

Other ways to debug vdsm:
- ps auxf - see vdsm child processes
- htop - you can enable thread names and use tree mode to see which vdsm
thread is consuming lot of cpu
- strace  - you can see which system calls vdsm is invoking
- gdb - you can attach to vdsm and get a python backtrace using py-bt

Nir

>
> Have a nice day
> --------------------------------
>
> pencc
>
>
> _______________________________________________
> Devel mailing list
> Devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20170814/b7dc4006/attachment.html>