8:05 a.m.
Hi,
This looks like something wrong with CI agents, in the part which scrubs
chroot we can find:
[2021-01-19T12:10:44.669Z] Finish: scrub ['chroot']
[2021-01-19T12:10:44.669Z] Traceback (most recent call last):
[2021-01-19T12:10:44.669Z] File "/dev/fd/6", line 1059, in <module>
[2021-01-19T12:10:44.669Z] exitStatus = main()
[2021-01-19T12:10:44.669Z] File
"/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line
93, in trace
[2021-01-19T12:10:44.669Z] result = func(*args, **kw)
[2021-01-19T12:10:44.669Z] File "/dev/fd/6", line 825, in main
[2021-01-19T12:10:44.669Z] result = run_command(options, args,
config_opts, commands, buildroot, state)
[2021-01-19T12:10:44.669Z] File
"/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line
93, in trace
[2021-01-19T12:10:44.669Z] result = func(*args, **kw)
[2021-01-19T12:10:44.670Z] File "/dev/fd/6", line 859, in run_command
[2021-01-19T12:10:44.670Z] commands.scrub(options.scrub)
[2021-01-19T12:10:44.670Z] File
"/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line
93, in trace
[2021-01-19T12:10:44.670Z] result = func(*args, **kw)
[2021-01-19T12:10:44.670Z] File
"/usr/lib/python3.6/site-packages/mockbuild/backend.py", line 125, in scrub
[2021-01-19T12:10:44.670Z] self.buildroot.delete()
[2021-01-19T12:10:44.670Z] File
"/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line
93, in trace
[2021-01-19T12:10:44.670Z] result = func(*args, **kw)
[2021-01-19T12:10:44.670Z] File
"/usr/lib/python3.6/site-packages/mockbuild/buildroot.py", line 869, in
delete
[2021-01-19T12:10:44.670Z] file_util.rmtree(self.basedir,
selinux=self.selinux)
[2021-01-19T12:10:44.670Z] File
"/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line
93, in trace
[2021-01-19T12:10:44.670Z] result = func(*args, **kw)
[2021-01-19T12:10:44.670Z] File
"/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 59, in
rmtree
[2021-01-19T12:10:44.670Z] rmtree(fullname, selinux=selinux,
exclude=exclude)
[2021-01-19T12:10:44.670Z] File
"/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line
93, in trace
[2021-01-19T12:10:44.670Z] result = func(*args, **kw)
[2021-01-19T12:10:44.670Z] File
"/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 59, in
rmtree
[2021-01-19T12:10:44.670Z] rmtree(fullname, selinux=selinux,
exclude=exclude)
[2021-01-19T12:10:44.670Z] File
"/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line
93, in trace
[2021-01-19T12:10:44.670Z] result = func(*args, **kw)
[2021-01-19T12:10:44.670Z] File
"/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 59, in
rmtree
[2021-01-19T12:10:44.670Z] rmtree(fullname, selinux=selinux,
exclude=exclude)
[2021-01-19T12:10:44.670Z] File
"/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line
93, in trace
[2021-01-19T12:10:44.670Z] result = func(*args, **kw)
[2021-01-19T12:10:44.670Z] File
"/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 59, in
rmtree
[2021-01-19T12:10:44.670Z] rmtree(fullname, selinux=selinux,
exclude=exclude)
[2021-01-19T12:10:44.670Z] File
"/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line
93, in trace
[2021-01-19T12:10:44.670Z] result = func(*args, **kw)
[2021-01-19T12:10:44.670Z] File
"/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 59, in
rmtree
[2021-01-19T12:10:44.670Z] rmtree(fullname, selinux=selinux,
exclude=exclude)
[2021-01-19T12:10:44.670Z] File
"/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line
93, in trace
[2021-01-19T12:10:44.670Z] result = func(*args, **kw)
[2021-01-19T12:10:44.670Z] File
"/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 68, in
rmtree
[2021-01-19T12:10:44.670Z] os.rmdir(path)
[2021-01-19T12:10:44.670Z] OSError: [Errno 16] Device or resource busy:
'/var/lib/mock/epel-8-x86_64-8e6e57af00668dc4b1252bf1bf4d754c-2645436/root/var/tmp/vdsm-storage/mount.file-512'
[2021-01-19T12:10:44.670Z] Scrub chroot took 10 seconds
Evgheni, could you please take a look at it?
Regards, Marcin
On 1/19/21 1:49 PM, Nir Soffer wrote:
We seem to have an issue in the CI, starting this week.
All tests pass, but creating coverage report fail:
+ generate_combined_coverage_report
+ pushd tests
~/tests ~
+ pwd
/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests
+ ls .coverage-gluster .coverage-lib .coverage-network .coverage-nose
.coverage-storage .coverage-virt
.coverage-gluster
.coverage-lib
.coverage-network
.coverage-nose
.coverage-storage
.coverage-virt
+ python3 -m coverage combine .coverage-gluster .coverage-lib
.coverage-network .coverage-nose .coverage-storage .coverage-virt
No usable data files
Coverage.py warning: Couldn't read data from
'/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-gluster':
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position
106: invalid start byte
Coverage.py warning: Couldn't read data from
'/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-lib':
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position
106: invalid start byte
Coverage.py warning: Couldn't read data from
'/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-network':
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position
106: invalid start byte
Coverage.py warning: Couldn't read data from
'/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-nose':
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position
106: invalid start byte
Coverage.py warning: Couldn't read data from
'/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-storage':
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position
106: invalid start byte
Coverage.py warning: Couldn't read data from
'/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-virt':
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position
106: invalid start byte
This looks like a bug in coverage, or maybe we use the wrong version
of this tool?
+ teardown
+ res=1
+ '[' 1 -ne 0 ']'
+ echo '*** err: 1'
*** err: 1
Additionally, we have this very strange error:
+ teardown_storage
+ python3 tests/storage/userstorage.py teardown
python3: can't open file 'tests/storage/userstorage.py': [Errno 2] No
such file or directory
This error is impossible, since tests/storage/userstorage.py is part
of the source.
Is it possible that some other code has deleted the source while a
build was running?
+ echo 'WARNING: Ingoring error while tearing down user storage'
WARNING: Ingoring error while tearing down user storage
We were careful not to fail the build because this issue, but it failed because
coverage report could not be created.
There are multiple instance of this issue:
-
https://jenkins.ovirt.org/job/vdsm_standard-check-patch/26107//artifact/c...
-
https://jenkins.ovirt.org/job/vdsm_standard-check-patch/26103//artifact/c...
-
https://jenkins.ovirt.org/job/vdsm_standard-check-patch/26102//artifact/c...
I think Eyal reported another instance, but I cannot find it now.
2 weeks ago we updated pytest and pytest-cov to latest version, but we did
not see this issue in the CI when we tested the change:
https://gerrit.ovirt.org/c/vdsm/+/112604
And CI was fine since then, so this may not be related, unless this is
a regression
in latest coverage or pytest-cov.
Nir