Hi, This looks like something wrong with CI agents, in the part which scrubs chroot we can find: [2021-01-19T12:10:44.669Z] Finish: scrub ['chroot'] [2021-01-19T12:10:44.669Z] Traceback (most recent call last): [2021-01-19T12:10:44.669Z] File "/dev/fd/6", line 1059, in <module> [2021-01-19T12:10:44.669Z] exitStatus = main() [2021-01-19T12:10:44.669Z] File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace [2021-01-19T12:10:44.669Z] result = func(*args, **kw) [2021-01-19T12:10:44.669Z] File "/dev/fd/6", line 825, in main [2021-01-19T12:10:44.669Z] result = run_command(options, args, config_opts, commands, buildroot, state) [2021-01-19T12:10:44.669Z] File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace [2021-01-19T12:10:44.669Z] result = func(*args, **kw) [2021-01-19T12:10:44.670Z] File "/dev/fd/6", line 859, in run_command [2021-01-19T12:10:44.670Z] commands.scrub(options.scrub) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace [2021-01-19T12:10:44.670Z] result = func(*args, **kw) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/backend.py", line 125, in scrub [2021-01-19T12:10:44.670Z] self.buildroot.delete() [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace [2021-01-19T12:10:44.670Z] result = func(*args, **kw) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/buildroot.py", line 869, in delete [2021-01-19T12:10:44.670Z] file_util.rmtree(self.basedir, selinux=self.selinux) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace [2021-01-19T12:10:44.670Z] result = func(*args, **kw) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 59, in rmtree [2021-01-19T12:10:44.670Z] rmtree(fullname, selinux=selinux, exclude=exclude) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace [2021-01-19T12:10:44.670Z] result = func(*args, **kw) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 59, in rmtree [2021-01-19T12:10:44.670Z] rmtree(fullname, selinux=selinux, exclude=exclude) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace [2021-01-19T12:10:44.670Z] result = func(*args, **kw) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 59, in rmtree [2021-01-19T12:10:44.670Z] rmtree(fullname, selinux=selinux, exclude=exclude) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace [2021-01-19T12:10:44.670Z] result = func(*args, **kw) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 59, in rmtree [2021-01-19T12:10:44.670Z] rmtree(fullname, selinux=selinux, exclude=exclude) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace [2021-01-19T12:10:44.670Z] result = func(*args, **kw) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 59, in rmtree [2021-01-19T12:10:44.670Z] rmtree(fullname, selinux=selinux, exclude=exclude) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace [2021-01-19T12:10:44.670Z] result = func(*args, **kw) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 68, in rmtree [2021-01-19T12:10:44.670Z] os.rmdir(path) [2021-01-19T12:10:44.670Z] OSError: [Errno 16] Device or resource busy: '/var/lib/mock/epel-8-x86_64-8e6e57af00668dc4b1252bf1bf4d754c-2645436/root/var/tmp/vdsm-storage/mount.file-512' [2021-01-19T12:10:44.670Z] Scrub chroot took 10 seconds Evgheni, could you please take a look at it? Regards, Marcin On 1/19/21 1:49 PM, Nir Soffer wrote:
We seem to have an issue in the CI, starting this week.
All tests pass, but creating coverage report fail:
+ generate_combined_coverage_report + pushd tests ~/tests ~ + pwd /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests + ls .coverage-gluster .coverage-lib .coverage-network .coverage-nose .coverage-storage .coverage-virt .coverage-gluster .coverage-lib .coverage-network .coverage-nose .coverage-storage .coverage-virt + python3 -m coverage combine .coverage-gluster .coverage-lib .coverage-network .coverage-nose .coverage-storage .coverage-virt No usable data files Coverage.py warning: Couldn't read data from '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-gluster': UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 106: invalid start byte Coverage.py warning: Couldn't read data from '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-lib': UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 106: invalid start byte Coverage.py warning: Couldn't read data from '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-network': UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 106: invalid start byte Coverage.py warning: Couldn't read data from '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-nose': UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 106: invalid start byte Coverage.py warning: Couldn't read data from '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-storage': UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 106: invalid start byte Coverage.py warning: Couldn't read data from '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-virt': UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 106: invalid start byte
This looks like a bug in coverage, or maybe we use the wrong version of this tool?
+ teardown + res=1 + '[' 1 -ne 0 ']' + echo '*** err: 1' *** err: 1
Additionally, we have this very strange error:
+ teardown_storage + python3 tests/storage/userstorage.py teardown python3: can't open file 'tests/storage/userstorage.py': [Errno 2] No such file or directory
This error is impossible, since tests/storage/userstorage.py is part of the source.
Is it possible that some other code has deleted the source while a build was running?
+ echo 'WARNING: Ingoring error while tearing down user storage' WARNING: Ingoring error while tearing down user storage
We were careful not to fail the build because this issue, but it failed because coverage report could not be created.
There are multiple instance of this issue: - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/26107//artifact/chec... - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/26103//artifact/chec... - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/26102//artifact/chec...
I think Eyal reported another instance, but I cannot find it now.
2 weeks ago we updated pytest and pytest-cov to latest version, but we did not see this issue in the CI when we tested the change: https://gerrit.ovirt.org/c/vdsm/+/112604
And CI was fine since then, so this may not be related, unless this is a regression in latest coverage or pytest-cov.
Nir