[ovirt-devel] [VDSM] Coverage reports

[David Caro Estevez]

On 11/27 20:57, Nir Soffer wrote:
> Hi all,
> 
> Thanks to Edward, we have now coverage reports in jenkins.
> 
> The way to access the report on jenkins is to use this url:
> http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/<build-number>/artifact/exported-artifacts/htmlcov/index.html
> 
> Here is an example, hopefully it will be accessible when you try:
> http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/648/artifact/exported-artifacts/htmlcov/index.html
> 
> Todo:
> 
> - Easy way to access the report from gerrit
>   It should be easy to add a link the coverage report in the comment added
>   by jenkins after a build finish.

Not really, the gerrit comments that jenkins sends can't be changed by the
build, that means that you can't put put there a non-static url.
There's a quite tricky way that can be done with groovy postscripts and
accessing the jenkins inner objects, but that's very likely to break on any
plugin upgrade and requires a bit of investigating with objects to change.

> 
> - Store the coverage reports for longer time, maybe a week?

If they are generated with the check patch, you always have the latest there
(stored as long as your builds). If you want longer archiving, might be a good
idea to generate them too in the check-merged.

> 
> - We have only 45% coverage instead of the minimum, 100%.
> 
>   Note that coverage of 25% can mean *no* code was run by the tests.
> The only code
>   running was the functions and class definitions. Here is an example:
>   http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/648/artifact/exported-artifacts/htmlcov/_home_jenkins_workspace_vdsm_master_check-patch-fc23-x86_64_vdsm_vdsm_storage_devicemapper_py.html
> 
>   Modules that were never imported during the tests have 0% coverage:
>   http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/648/artifact/exported-artifacts/htmlcov/_home_jenkins_workspace_vdsm_master_check-patch-fc23-x86_64_vdsm_vdsm_storage_hsm_py.html
> 
> - coverage creates lot of useless noise in the jenkins logs, need to slicense
>   this output.
>   http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/648/consoleFull
>   I did not find a way to do this in nosetests, may need hacking
>   nose coverage plugin.
> 
> - The report includes only the tests in the tests directory.
> 
>    We have additional tests in lib/vdsm/infra/* which are not
> included. We should
>    move these to the tests directory.
> 
> - The report is using absolute paths, but we like shorter relative paths.
> 
>   I don't see a way to configure nosetests or coverage to generate
> relative paths.
>   May need hacking of the generated html/json in htmlcov.

If you find it please tell me, I have the same issue with other projects, I'll
do if I discover a way :)

> 
> - Add "make coverage" target for running coverage locally
> 
> - An easy way to enable coverage for the functional tests or for running
>   a single test module.
> 
>   Can be done using nosetests --cover* options. Should be documented in
>   tests/README, and maybe automated using a script or Makefile.
>   When running locally, one would like to have the script open the report
>   in the browser automatically:
>   xdg-open tests/htmlcov/index.html
> 
> - An easy way to enable coverage when testing flows in vdsm
> 
>   Petr sent a patch for enabling coverage using vdsm.conf:
>   https://gerrit.ovirt.org/49168
>   We discussed adding vdsm-coverage package that will make it easy to setup
> 
> Nir

-- 
David Caro

Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D

Tel.: +420 532 294 605
Email: dcaro at redhat.com
IRC: dcaro|dcaroest@{freenode|oftc|redhat}
Web: www.redhat.com
RHT Global #: 82-62605
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20151127/2c21366e/attachment.sig>