<div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Mon, Aug 14, 2017 at 4:26 PM pengyixiang <<a href="mailto:yxpengi386@163.com">yxpengi386@163.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="font-family:"\005fae\008f6f\0096c5\009ed1";font-size:14px;color:#000000;line-height:1.7">
<div><span style="font-family:Verdana,微软雅黑,宋体,sans-serif;line-height:23.8px">hello everyone!</span><br style="font-family:Verdana,微软雅黑,宋体,sans-serif;line-height:23.8px"><div id="m_4967671231265583925isForwardContent" style="font-family:Verdana,微软雅黑,宋体,sans-serif;line-height:23.8px"><p> Now my main way to learn and debug vdsm is to write pdb code to vdsm and restart vdsmd.service, dose it have a better way? <span style="line-height:23.8px">As some ide that is more </span><span style="line-height:23.8px">convenient to</span><span style="line-height:23.8px"> add</span><span style="line-height:23.8px"> </span><span class="m_4967671231265583925dictTag" style="line-height:23.8px"></span><span style="line-height:23.8px">breaking point、step into, etc. </span></p></div></div></div></div></blockquote><div><br></div><div>The best way to debug vdsm is to write good tests. When you fail to write tests</div><div>or in untested areas, your best way is to use logs.</div><div><br></div><div>Stopping vdsm in the debugger may cause engine to fence it, and in general </div><div>stopping entire multi threaded program like vdsm is not very useful.</div><div><br></div><div>Another way to debug is via manhole - you can get a python shell inside vdsm,</div><div>and poke in its internals.</div><div><br></div><div>To use manhole install it:</div><div><br></div><div># pip install -U manhole</div><div><br></div><div>And enable using vdsm dropin conf - create a file:<br></div><div><br></div><div><div># cat /etc/vdsm/vdsm.conf.d/manhole.conf </div><div>[devel]</div><div># Enable manhole debugging service (requires manhole package).</div><div>manhole_enable = true</div></div><div><br></div><div>And restart vdsm.</div><div><br></div><div>To open a python shell inside vdsm you can use netcat, but socat is</div><div>much better because of the readline integration.</div><div><br></div><div># socat readline unix-connect:/run/vdsm/vdsmd.manhole<br></div><div><div><br></div><div>######### ProcessID=21004, ThreadID=139830629488384 #########</div><div>File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap</div><div> self.__bootstrap_inner()</div><div>File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner</div><div> self.run()</div><div>File: "/usr/lib/python2.7/site-packages/manhole.py", line 225, in run</div><div> self.handle(self.client, self.locals)</div><div>File: "/usr/lib/python2.7/site-packages/manhole.py", line 268, in handle</div><div> run_repl(locals)</div><div>File: "/usr/lib/python2.7/site-packages/manhole.py", line 313, in run_repl</div><div> dump_stacktraces()</div><div>File: "/usr/lib/python2.7/site-packages/manhole.py", line 578, in dump_stacktraces</div><div> for filename, lineno, name, line in traceback.extract_stack(stack):</div></div><div><br></div><div>[--snipped stacktrace of all threads--]</div><div><br></div><div><div>Python 2.7.5 (default, May 3 2017, 07:55:04) </div><div>[GCC 4.8.5 20150623 (Red Hat 4.8.5-14)] on linux2</div><div>Type "help", "copyright", "credits" or "license" for more information.</div><div>(ManholeConsole)</div><div>>>> </div></div><div><br></div><div>Checking your local namespace:</div><div><br></div><div><div>>>> dir()</div><div>['__builtins__', 'cif', 'dump_stacktraces', 'irs', 'os', 'socket', 'sys', 'traceback']</div></div><div><br></div><div>irs is the storage endpoint, you can look into it:</div><div><br></div><div><div>>>> irs._obj.domainMonitor._monitors</div><div>{u'b9cda581-f9ac-4650-83e0-17d00d7dfc75': <vdsm.storage.monitor.MonitorThread object at 0x3ecd6d0>, u'd4ad5a78-1d75-457b-8caf-400d2d7917a6': <vdsm.storage.monitor.MonitorThread object at 0x3e38890>, u'd6e4a622-bd31-4d8f-904d-1e26b7286757': <vdsm.storage.monitor.MonitorThread object at 0x3f2bf50>, u'6ffbc483-0031-403a-819b-3bb2f0f8de0a': <vdsm.storage.monitor.MonitorThread object at 0x3f2b5d0>, u'373e8c55-283f-41d4-8433-95c1ef1bbd1a': <vdsm.storage.monitor.MonitorThread object at 0x3ecde10>, u'2770b49f-79d7-4b7c-9d39-03e988e8bc4d': <vdsm.storage.monitor.MonitorThread object at 0x3f37690>}</div></div><div><br></div><div><div>>>> irs._obj.domainMonitor._monitors['d6e4a622-bd31-4d8f-904d-1e26b7286757'].status.readDelay</div><div>0.000905336</div></div><div><br></div><div><div>>>> irs._obj.domainMonitor._checker._loop._scheduled</div><div>[<Timer when=4920799.730000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2af6110> at 0x2396650>, <Timer when=4920799.750000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2afc890> at 0x2396d50>, <Timer when=4920800.690000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2b28890> at 0x2396590>, <Timer when=4920799.770000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2b21c90> at 0x2331a90>, <Timer when=4920800.620000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2b21d90> at 0x2a997d0>, <Timer when=4920800.800000 callback=<vdsm.storage.asyncutils.LoopingCall object at 0x2b9a9d0> at 0x23967d0>]</div></div><div><br></div><div><div><div>>>> irs._obj.domainMonitor._checker._loop._scheduled[0]._callback._callback</div><div><bound method DirectioChecker._check of <DirectioChecker /rhev/data-center/mnt/dumbo.tlv.redhat.com:_voodoo_41/2770b49f-79d7-4b7c-9d39-03e988e8bc4d/dom_md/metadata running next_check=4920929.73 at 0x2af6050>></div></div></div><div><br></div><div>dir(obj) is your best friend inside the manhole.</div><div><br></div><div>Other ways to debug vdsm:</div><div>- ps auxf - see vdsm child processes</div><div>- htop - you can enable thread names and use tree mode to see which vdsm thread is consuming lot of cpu</div><div>- strace - you can see which system calls vdsm is invoking</div><div>- gdb - you can attach to vdsm and get a python backtrace using py-bt</div><div><br></div><div>Nir</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:"\005fae\008f6f\0096c5\009ed1";font-size:14px;color:#000000;line-height:1.7"><div><div id="m_4967671231265583925isForwardContent" style="font-family:Verdana,微软雅黑,宋体,sans-serif;line-height:23.8px"><p><br></p><pre class="m_4967671231265583925moz-signature" cols="72" style="font-family:Verdana,微软雅黑,宋体,sans-serif!important">Have a nice day<br>--------------------------------<br></pre><p></p><p>pencc</p></div></div>
<div><br></div><div id="m_4967671231265583925ntes-pcmail-signature" style="font-family:'\005fae\008f6f\0096c5\009ed1'"><font style="padding:0;margin:0"> </font>
</div><br>
</div>
</div>
_______________________________________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/devel</a></blockquote></div></div>