Hi, This looks like something wrong with CI agents, in the part which scrubs chroot we can find: [2021-01-19T12:10:44.669Z] Finish: scrub ['chroot'] [2021-01-19T12:10:44.669Z] Traceback (most recent call last): [2021-01-19T12:10:44.669Z] File "/dev/fd/6", line 1059, in <module> [2021-01-19T12:10:44.669Z] exitStatus = main() [2021-01-19T12:10:44.669Z] File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace [2021-01-19T12:10:44.669Z] result = func(*args, **kw) [2021-01-19T12:10:44.669Z] File "/dev/fd/6", line 825, in main [2021-01-19T12:10:44.669Z] result = run_command(options, args, config_opts, commands, buildroot, state) [2021-01-19T12:10:44.669Z] File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace [2021-01-19T12:10:44.669Z] result = func(*args, **kw) [2021-01-19T12:10:44.670Z] File "/dev/fd/6", line 859, in run_command [2021-01-19T12:10:44.670Z] commands.scrub(options.scrub) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace [2021-01-19T12:10:44.670Z] result = func(*args, **kw) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/backend.py", line 125, in scrub [2021-01-19T12:10:44.670Z] self.buildroot.delete() [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace [2021-01-19T12:10:44.670Z] result = func(*args, **kw) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/buildroot.py", line 869, in delete [2021-01-19T12:10:44.670Z] file_util.rmtree(self.basedir, selinux=self.selinux) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace [2021-01-19T12:10:44.670Z] result = func(*args, **kw) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 59, in rmtree [2021-01-19T12:10:44.670Z] rmtree(fullname, selinux=selinux, exclude=exclude) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace [2021-01-19T12:10:44.670Z] result = func(*args, **kw) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 59, in rmtree [2021-01-19T12:10:44.670Z] rmtree(fullname, selinux=selinux, exclude=exclude) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace [2021-01-19T12:10:44.670Z] result = func(*args, **kw) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 59, in rmtree [2021-01-19T12:10:44.670Z] rmtree(fullname, selinux=selinux, exclude=exclude) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace [2021-01-19T12:10:44.670Z] result = func(*args, **kw) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 59, in rmtree [2021-01-19T12:10:44.670Z] rmtree(fullname, selinux=selinux, exclude=exclude) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace [2021-01-19T12:10:44.670Z] result = func(*args, **kw) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 59, in rmtree [2021-01-19T12:10:44.670Z] rmtree(fullname, selinux=selinux, exclude=exclude) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace [2021-01-19T12:10:44.670Z] result = func(*args, **kw) [2021-01-19T12:10:44.670Z] File "/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 68, in rmtree [2021-01-19T12:10:44.670Z] os.rmdir(path) [2021-01-19T12:10:44.670Z] OSError: [Errno 16] Device or resource busy: '/var/lib/mock/epel-8-x86_64-8e6e57af00668dc4b1252bf1bf4d754c-2645436/root/var/tmp/vdsm-storage/mount.file-512' [2021-01-19T12:10:44.670Z] Scrub chroot took 10 seconds Evgheni, could you please take a look at it? Regards, Marcin On 1/19/21 1:49 PM, Nir Soffer wrote: > We seem to have an issue in the CI, starting this week. > > All tests pass, but creating coverage report fail: > > + generate_combined_coverage_report > + pushd tests > ~/tests ~ > + pwd > /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests > + ls .coverage-gluster .coverage-lib .coverage-network .coverage-nose > .coverage-storage .coverage-virt > .coverage-gluster > .coverage-lib > .coverage-network > .coverage-nose > .coverage-storage > .coverage-virt > + python3 -m coverage combine .coverage-gluster .coverage-lib > .coverage-network .coverage-nose .coverage-storage .coverage-virt > No usable data files > Coverage.py warning: Couldn't read data from > '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-gluster': > UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position > 106: invalid start byte > Coverage.py warning: Couldn't read data from > '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-lib': > UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position > 106: invalid start byte > Coverage.py warning: Couldn't read data from > '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-network': > UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position > 106: invalid start byte > Coverage.py warning: Couldn't read data from > '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-nose': > UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position > 106: invalid start byte > Coverage.py warning: Couldn't read data from > '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-storage': > UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position > 106: invalid start byte > Coverage.py warning: Couldn't read data from > '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-virt': > UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position > 106: invalid start byte > > This looks like a bug in coverage, or maybe we use the wrong version > of this tool? > > + teardown > + res=1 > + '[' 1 -ne 0 ']' > + echo '*** err: 1' > *** err: 1 > > Additionally, we have this very strange error: > > + teardown_storage > + python3 tests/storage/userstorage.py teardown > python3: can't open file 'tests/storage/userstorage.py': [Errno 2] No > such file or directory > > This error is impossible, since tests/storage/userstorage.py is part > of the source. > > Is it possible that some other code has deleted the source while a > build was running? > > + echo 'WARNING: Ingoring error while tearing down user storage' > WARNING: Ingoring error while tearing down user storage > > We were careful not to fail the build because this issue, but it failed because > coverage report could not be created. > > There are multiple instance of this issue: > - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/26107//artifact/c... > - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/26103//artifact/c... > - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/26102//artifact/c... > > I think Eyal reported another instance, but I cannot find it now. > > 2 weeks ago we updated pytest and pytest-cov to latest version, but we did > not see this issue in the CI when we tested the change: > https://gerrit.ovirt.org/c/vdsm/+/112604 > > And CI was fine since then, so this may not be related, unless this is > a regression > in latest coverage or pytest-cov.