[ovirt-devel] ovirt-engine-sdk-python too slow

Nir Soffer nsoffer at redhat.com
Wed Nov 25 17:45:18 UTC 2015


$ ./profile-stats -c myscript.prof

Wed Nov 25 10:40:11 2015    myscript.prof

         7892315 function calls (7891054 primitive calls) in 7.940 seconds

   Ordered by: internal time
   List reduced from 1518 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     9086    2.693    0.000    6.706    0.001 inspect.py:247(getmembers)
  1952494    1.394    0.000    1.880    0.000 inspect.py:59(isclass)
     9092    1.030    0.000    1.030    0.000 {dir}
  1952642    0.600    0.000    0.600    0.000 {getattr}
  1972765    0.504    0.000    0.504    0.000 {isinstance}
        3    0.334    0.111    0.334    0.111 {method 'perform' of
'pycurl.Curl' objects}
  1883918    0.284    0.000    0.284    0.000 {method 'append' of 'list'
objects}
     9087    0.221    0.000    0.221    0.000 {method 'sort' of 'list'
objects}
     9051    0.172    0.000    6.911    0.001
reflectionhelper.py:51(isModuleMember)
        1    0.124    0.124    0.354    0.354 errors.py:17(<module>)
        1    0.088    0.088    0.230    0.230 params.py:8(<module>)
     8879    0.070    0.000    6.998    0.001 params.py:367(__setattr__)
        1    0.047    0.047    5.182    5.182 api.py:23(<module>)
        1    0.025    0.025    4.743    4.743 brokers.py:22(<module>)
        1    0.023    0.023    0.030    0.030
connectionspool.py:17(<module>)
        1    0.022    0.022    0.053    0.053
lxml.etree.pyx:1(PyMODINIT_FUNC PyInit_etree(void))
      118    0.019    0.000    4.684    0.040 params.py:45277(__init__)
        5    0.015    0.003    0.024    0.005 {built-in method strptime}
        1    0.012    0.012    0.013    0.013 socket.py:45(<module>)
       10    0.011    0.001    0.015    0.002 collections.py:288(namedtuple)

So it is not the classes, it is the code inspecting them on import.

Nir

On Wed, Nov 25, 2015 at 7:49 AM, John Hunter <zhjwpku at gmail.com> wrote:

> Hi Nir,
>
> Attachment is my script and its profile.
> Thanks a lot about your help!
>
> On Wed, Nov 25, 2015 at 5:49 AM, Nir Soffer <nsoffer at redhat.com> wrote:
>
>> On Tue, Nov 24, 2015 at 3:49 PM, John Hunter <zhjwpku at gmail.com> wrote:
>>
>>>
>>>
>>> On Tue, Nov 24, 2015 at 9:15 PM, Juan Hernández <jhernand at redhat.com>
>>> wrote:
>>>
>>>> On 11/24/2015 01:40 PM, John Hunter wrote:
>>>> > Hi,
>>>> >
>>>> > On Tue, Nov 24, 2015 at 5:18 PM, Oved Ourfali <oourfali at redhat.com
>>>> > <mailto:oourfali at redhat.com>> wrote:
>>>> >
>>>> >     Hi
>>>> >
>>>> >     I discussed it with Juan (cc-ed).
>>>> >
>>>> >     There used to be a bug in the JDBC authenticion extension that
>>>> >     artificially delayed RESTAPI responses by 5 seconds:
>>>> >
>>>> >       brute force prevention login delay should not be applied to
>>>> successful
>>>> >     login requests
>>>> >       https://bugzilla.redhat.com/1255814
>>>> >
>>>> >     That matches the description of the issue, but in theory it has
>>>> been
>>>> >     fixed. I would suggest him to check that he is using the right
>>>> version
>>>> >     of the extension.
>>>> >
>>>> > I did not use the extension ovirt-engine-extension-aaa-jdbc, and I
>>>> don't
>>>> > think this bug matches my problem, because even there is only one line
>>>> > in the python script, it still cost like 3 seconds, I don't think
>>>> this is a
>>>> > reasonable time as when I import other package, it cost almost no
>>>> time.
>>>> >
>>>> > Can you explain why this import line costs so much time?
>>>> >
>>>>
>>>> If you are using 3.6 then you are using ovirt-engine-extension-aaa-jdbc,
>>>> as it is enabled by default, but looks like it isn't related to your
>>>> problem.
>>>>
>>>> That line takes a long time to execute because it has to process two
>>>> large Python modules: the "params" module that contains a class per each
>>>> type used by the API (393 classes) and the "brokers" module that
>>>> contains a class per each resource used by the API (358 classes). That
>>>> makes a total of 751 classes. In my environment it takes 0.9 seconds,
>>>> approx. You may want to use the python profile in your environment and
>>>> share the results:
>>>>
>>>> $ cat > profile.py <<.
>>>> import cProfile
>>>> cProfile.run("from ovirtsdk.api import API")
>>>> .
>>>>
>>>> $ python profile.py
>>>>
>>>> I won't be surprised to see this taking those 3 seconds in a slower
>>>> environment.
>>>>
>>>> But even if this takes those 3 seconds it shouldn't be a big problem,
>>>> because you shouldn't be running that "from ... import ..." line
>>>> frequently. Your program should do this once only, when it starts.
>>>>
>>>> Assume that I have two functions to implement, one is to list all the
>>> vms belong
>>> to the user, and the other is to retrieve one vm's virt-viewer
>>> connection file, as
>>> far as I can see, I have to write two python scripts and import the
>>> ovirtsdk.api in both
>>> scripts, each script has to take the 3 seconds :(
>>>
>>
>> No, you have two functions, which can be in the same module, or in
>> different modules, depending on how you want to organize your code.
>>
>> Python imports are done only once, on the first time you import a module
>> anywhere. The module is stored in sys.modules, and the next import fetch
>> the module object from sys.modules.
>>
>> Can you share a profile of a simple script importing the Python sdk?
>>
>> Do:
>>
>>     python -m cProfile -o myscript.prof myscript.py
>>
>> The profile will be stored in myscript.prof.
>>
>> For viewing the profile, I recommend to use the profile-stats tool:
>> https://github.com/oVirt/vdsm/blob/master/contrib/profile-stats
>>
>>
>>>
>>> How can I run the "from ... import ..." just once ?
>>>
>>>
>>>> >
>>>> >     In addition we also know that retrieving large lists of objects
>>>> with the
>>>> >     SDK is slow:
>>>> >
>>>> >        [RFE][performance] - generate large scale list running to slow.
>>>> >        https://bugzilla.redhat.com/1221238
>>>> >
>>>> >     We don't have a solution for that yet.
>>>> >
>>>> >     CC-ing Juan in case you have additional questions.
>>>> >
>>>> >
>>>> >     On Mon, Nov 23, 2015 at 11:27 AM, John Hunter <zhjwpku at gmail.com
>>>> >     <mailto:zhjwpku at gmail.com>> wrote:
>>>> >
>>>> >         Hi guys,
>>>> >
>>>> >         I am using the ovirt-engine-sdk-python to communicate with the
>>>> >         ovirt-engine,
>>>> >         I am ok to list the vms but the processing time is too long,
>>>> >         like 4.5 seconds,
>>>> >         and this line:
>>>> >             from ovirtsdk.api import API
>>>> >         take almost 3 seconds.
>>>> >
>>>> >         This seems a little bit longer than I expected it to be, so I
>>>> am
>>>> >         asking is there
>>>> >         a quicker way to communicate with the ovirt-engine?
>>>> >
>>>>
>>>> --
>>>> Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta
>>>> 3ºD, 28016 Madrid, Spain
>>>> Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat
>>>> S.L.
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards
>>> Junwang Zhao
>>> Department of Computer Science &Technology
>>> Peking University
>>> Beijing, 100871, PRC
>>>
>>> _______________________________________________
>>> Devel mailing list
>>> Devel at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>
>>
>>
>
>
> --
> Best regards
> Junwang Zhao
> Department of Computer Science &Technology
> Peking University
> Beijing, 100871, PRC
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20151125/4b3e0ec2/attachment-0001.html>


More information about the Devel mailing list