We seem to have an issue in the CI, starting this week. All tests pass, but creating coverage report fail: + generate_combined_coverage_report + pushd tests ~/tests ~ + pwd /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests + ls .coverage-gluster .coverage-lib .coverage-network .coverage-nose .coverage-storage .coverage-virt .coverage-gluster .coverage-lib .coverage-network .coverage-nose .coverage-storage .coverage-virt + python3 -m coverage combine .coverage-gluster .coverage-lib .coverage-network .coverage-nose .coverage-storage .coverage-virt No usable data files Coverage.py warning: Couldn't read data from '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-gluster': UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 106: invalid start byte Coverage.py warning: Couldn't read data from '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-lib': UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 106: invalid start byte Coverage.py warning: Couldn't read data from '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-network': UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 106: invalid start byte Coverage.py warning: Couldn't read data from '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-nose': UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 106: invalid start byte Coverage.py warning: Couldn't read data from '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-storage': UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 106: invalid start byte Coverage.py warning: Couldn't read data from '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-virt': UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 106: invalid start byte This looks like a bug in coverage, or maybe we use the wrong version of this tool? + teardown + res=1 + '[' 1 -ne 0 ']' + echo '*** err: 1' *** err: 1 Additionally, we have this very strange error: + teardown_storage + python3 tests/storage/userstorage.py teardown python3: can't open file 'tests/storage/userstorage.py': [Errno 2] No such file or directory This error is impossible, since tests/storage/userstorage.py is part of the source. Is it possible that some other code has deleted the source while a build was running? + echo 'WARNING: Ingoring error while tearing down user storage' WARNING: Ingoring error while tearing down user storage We were careful not to fail the build because this issue, but it failed because coverage report could not be created. There are multiple instance of this issue: - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/26107//artifact/chec... - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/26103//artifact/chec... - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/26102//artifact/chec... I think Eyal reported another instance, but I cannot find it now. 2 weeks ago we updated pytest and pytest-cov to latest version, but we did not see this issue in the CI when we tested the change: https://gerrit.ovirt.org/c/vdsm/+/112604 And CI was fine since then, so this may not be related, unless this is a regression in latest coverage or pytest-cov. Nir