OST Failure - Weekly update [09/03/2018-16/03/2018]
by Dafna Ron
Hello,
I would like to update on this week's failures and OST current status.
We had a very eventful week with 2 regressions identified by OST
1. On 11-03-2018 the CI team reported a regression in project ovirt-engine.
The change reported was https://gerrit.ovirt.org/#/c/88738/ - db: add func
to turn table columns to empty string. This caused a failure in both master
and 4.2 in OST but was also identified to be an issue in 4.1.
The fix for all 3 versions can be tracked here:
https://bugzilla.redhat.com/show_bug.cgi?id=1554377 and was submitted on
15-03-2018.
I hope that in future we would resolve such regression much faster as it
took us 5 days to resolve this issue but I would also like to thank all
that helped to resolve it once the issue was investigated more deeply.
2. on 13-03-2018 we found a regression in otopi:
*core: Use python3 when possible- https://gerrit.ovirt.org/#/c/87276/
<https://gerrit.ovirt.org/#/c/87276/>*
*The fix was* submitted *on the same day **https://gerrit.ovirt.org/88931
<https://gerrit.ovirt.org/88931> - **spec: Package otopi-functions*
We also had 1 configuration related issue which was fixed by the CI team
and we had failed build-artifacts, mostly in fc27. this was due to the
change in fedora to not allow the use of yum and had to be solved by the
projects making minor patch changes.
*Below you can see the chart for this week's resolved issues but cause of
failure:*Code = regression of working components/functionalities
Configurations = package related issues
Other = failed build artifacts
Infra = infrastructure/OST/Lago related issues
*Below is a chart of resolved failures based on ovirt version*
*Below is a chart showing failures by suite type: *
Thank you,
Dafna
6 years, 8 months
[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build # 223 - Failure!
by jenkins＠jenkins.phx.ovirt.org
------=_Part_248_224867011.1521170625791
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/223/
Build Number: 223
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #223
[Michal Skrivanek] master: add USB to the sanity VM
[Sandro Bonazzola] mirrors-reposync: add qemu-ev testing repo
[Daniel Belenky] Modify mergeoptions in parser utils to merge runif
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: 002_bootstrap.add_hosts
Error Message:
None != True after 900 seconds
Stack Trace:
Traceback (most recent call last):
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 230, in add_hosts
add_hosts_4(prefix)
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 274, in add_hosts_4
testlib.assert_true_within(_host_is_up_4, timeout=15*60)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 263, in assert_true_within
assert_equals_within(func, True, timeout, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 237, in assert_equals_within
'%s != %s after %s seconds' % (res, value, timeout)
AssertionError: None != True after 900 seconds
------=_Part_248_224867011.1521170625791--
6 years, 8 months
[ OST Failure Report ] [ oVirt Master (cockpit-ovirt) ] [ 15-03-2018 ] [ 098_ovirt_provider_ovn.use_ovn_provider ]
by Dafna Ron
Hi,
We have a failure on master for test
098_ovirt_provider_ovn.use_ovn_provider in project cockpit-ovirt.
This seems to be a race because object is locked. also, the actual failure
is logged as WARN and not ERROR.
I don't think the patch is actually related to the failure but I think the
test should be fixed.
can you please review to make sure we do not have an actual regression and
let me know if we need to open a bz to fix the test?
*Link and headline of suspected patches: *
*https://gerrit.ovirt.org/#/c/89020/2
<https://gerrit.ovirt.org/#/c/89020/2> - *
*wizard: Enable scroll on start page for low-res screensLink to
Job:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/6374
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/6374>Link to
all
logs:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/6374/a...
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/6374/artifacts>(Relevant)
error snippet from the log: <error>2018-03-15 10:05:00,160-04 DEBUG
[org.ovirt.engine.core.sso.servlets.OAuthTokenInfoServlet] (default
task-10) [] Sending json response2018-03-15 10:05:00,160-04 DEBUG
[org.ovirt.engine.core.sso.utils.TokenCleanupUtility] (default task-10) []
Not cleaning up expired tokens2018-03-15 10:05:00,169-04 INFO
[org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-90) [789edb23] Lock
Acquired to object
'EngineLock:{exclusiveLocks='[c38a67ec-0b48-4e6f-be85-70c700df5483=PROVIDER]',
sharedLocks=''}'2018-03-15 10:05:00,184-04 INFO
[org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-90) [789edb23] Running
command: SyncNetworkProviderCommand internal: true.2018-03-15
10:05:00,228-04 DEBUG
[org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall]
(default task-13) [e1328379-17b7-49f8-beb2-cf8331784828] Compiled stored
procedure. Call string is [{call getdcidbyexternalnetworkid(?)}]2018-03-15
10:05:00,228-04 DEBUG
[org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall]
(default task-13) [e1328379-17b7-49f8-beb2-cf8331784828] SqlCall for
procedure [GetDcIdByExternalNetworkId] compiled2018-03-15 10:05:00,229-04
DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
(default task-13) [e1328379-17b7-49f8-beb2-cf8331784828] method: runQuery,
params: [GetAllExternalNetworksOnProvider,
IdQueryParameters:{refresh='false', filtered='false'}], timeElapsed:
353ms2018-03-15 10:05:00,239-04 INFO
[org.ovirt.engine.core.bll.network.dc.AddNetworkCommand] (default task-13)
[e1328379-17b7-49f8-beb2-cf8331784828] Failed to Acquire Lock to object
'EngineLock:{exclusiveLocks='[network_1=NETWORK,
c38a67ec-0b48-4e6f-be85-70c700df5483=PROVIDER]', sharedLocks=''}'2018-03-15
10:05:00,239-04 WARN
[org.ovirt.engine.core.bll.network.dc.AddNetworkCommand] (default task-13)
[e1328379-17b7-49f8-beb2-cf8331784828] Validation of action 'AddNetwork'
failed for user admin@internal-authz. Reasons:
VAR__TYPE__NETWORK,VAR__ACTION__ADD,ACTION_TYPE_FAILED_PROVIDER_LOCKED,$providerId
c38a67ec-0b48-4e6f-be85-70c700df54832018-03-15 10:05:00,240-04 DEBUG
[org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
(default task-13) [e1328379-17b7-49f8-beb2-cf8331784828] method: runAction,
params: [AddNetwork,
AddNetworkStoragePoolParameters:{commandId='61b365ec-27c1-49af-ad72-f907df8befcd',
user='null', commandType='Unknown'}], timeElapsed: 10ms2018-03-15
10:05:00,250-04 ERROR
[org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default
task-13) [] Operation Failed: [Cannot add Network. Related operation on
provider with the id c38a67ec-0b48-4e6f-be85-70c700df5483 is currently in
progress. Please try again later.]2018-03-15 10:05:00,254-04 DEBUG
[org.ovirt.engine.core.utils.servlet.LocaleFilter] (default task-14) []
Incoming locale 'en-US'. Filter determined locale to be 'en-US'2018-03-15
10:05:00,254-04 DEBUG
[org.ovirt.engine.core.sso.servlets.OAuthTokenServlet] (default task-14) []
Entered OAuthTokenServlet Query String: null, Parameters : password = ***,
grant_type = password, scope = ovirt-app-api ovirt-ext=token-info:validate,
username = admin@internal, </error>*
6 years, 8 months
Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #1373
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...>
Changes:
[Eyal Edri] Run all nightly suites 3 times a day
[Sandro Bonazzola] system-tests: add he-basic-iscsi-suite-master
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...>
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision f395591f18d0a3b6ae3ff8daaafd93321b7d61f3 (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f f395591f18d0a3b6ae3ff8daaafd93321b7d61f3
Commit message: "system-tests: add he-basic-iscsi-suite-master"
> git rev-list --no-walk 26142503ea0774cd18e740630cd68c7ab86d9f83 # timeout=10
[system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins4149197632192240639.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf
Checking if mirror needs a resync
Traceback (most recent call last):
File "/usr/bin/reposync", line 343, in <module>
main()
File "/usr/bin/reposync", line 175, in main
my.doRepoSetup()
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
return self._getRepos(thisrepo, True)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
self._repos.doSetup(thisrepo)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
dl = repo._async and repo._commonLoadRepoXML(repo)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1474, in _commonLoadRepoXML
if self._latestRepoXML(local):
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1443, in _latestRepoXML
oxml = self._saveOldRepoXML(local)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1300, in _saveOldRepoXML
shutil.copy2(local, old_local)
File "/usr/lib64/python2.7/shutil.py", line 131, in copy2
copystat(src, dst)
File "/usr/lib64/python2.7/shutil.py", line 98, in copystat
os.utime(dst, (st.st_atime, st.st_mtime))
OSError: [Errno 2] No such file or directory: '/home/jenkins/mirrors_cache/fedora-updates-fc24/repomd.xml.old.tmp'
Build step 'Execute shell' marked build as failure
6 years, 8 months
