[ovirt-devel] [VDSM] Profiling results - 144 cores, 144 vms
Hi all,
I noticed that Eldad posted profiling result from 144 cores / 144 vms run:
See https://bugzilla.redhat.com/attachment.cgi?id=1128028
Looking at the profile bellow:
1. Why _rtnl_link_alloc_cache is so expensive?
called only 389 times, but the most expensive call
2. Creating DomainDescriptor seems expensive - is this xml.minidom?
3. guestagent.py:84(filt) is very expensive - we probbaly want to backport
7091bde guestagent: Speed up xml character filtering, can shave about
17% of cpu time in this run.
4. pthread.py function will disapper from the profile if we replace
pthreading with cthreading:
https://github.com/nirs/cthreading
5. Why do we see xmlrpc at the top of the profile? Is this the cost
of mom calls?
Eldad, can you repeat the same test with current master?
Nir
----
Thu Feb 18 00:50:24 2016 vdsm_144vms_1800sec_2.prof
29858105 function calls (34034631 primitive calls) in 1049.574 seconds
Ordered by: internal time
List reduced from 2420 to 20 due to restriction <20>
ncalls tottime percall cumtime percall filename:lineno(function)
389 64.937 0.167 64.941 0.167
link.py:133(_rtnl_link_alloc_cache)
598438 64.176 0.000 87.888 0.000
domain_descriptor.py:23(DomainDescriptor)
1535528 38.922 0.001 176.177 0.000 guestagent.py:84(filt)
57106 31.545 0.001 31.560 0.001
decoder.py:371(JSONDecoder.raw_decode)
168463 25.961 0.234 119.004 0.001
xmlrpclib.py:721(Marshaller.dump_struct)
4312452 25.095 0.000 77.012 0.000 utf_8.py:15(decode)
7341795 21.934 0.000 41.507 0.000 string.py:511(replace)
28881 18.092 0.001 19.921 0.001 pthread.py:133(Cond.timedwait)
386282 17.509 0.000 17.509 0.000 pthread.py:95(Lock.lock)
46277 16.798 0.000 45.280 0.001
guestagent.py:254(GuestAgent._handleMessage)
48090 16.298 0.000 37.809 0.001
expatbuilder.py:743(ExpatBuilderNS.start_element_handler)
2447266 14.128 0.000 55.635 0.000 httplib.py:331(HTTPResponse)
339514 13.689 0.014 20.629 0.000 password.py:69(_walk)
78549 12.450 0.000 12.635 0.000 minidom.py:344(Attr.__init__)
20117 12.381 0.001 20.961 0.001 vm.py:181(SetLinkAndNetworkError)
14470 12.224 0.001 12.802 0.001 xmlbuilder.py:1(<module>)
21404 11.527 0.001 36.225 0.002 vmstats.py:240(disks)
87612 11.441 0.000 35.701 0.000
sampling.py:59(InterfaceSample.readIfaceStat)
39924 11.323 0.000 23.047 0.001
__init__.py:242(LogRecord.__init__)
14509 10.816 0.001 96.236 0.007 xmlbuilder.py:12(Options)