[ OST Failure Report ] [ oVirt Master (ovirt-engine-nodejs-modules) ]
[ 27-02-2019 ] [ 002_bootstrap.add_vm2_lease ]
by Dafna Ron
Hi,
We have a failure on the project in basic suite, master branch. The recent
failure was in patch:
https://gerrit.ovirt.org/#/c/98087/1 - Add pre-seed for ovirt-web-ui
CQ is pointing at the below as the root cause (which was merged a while
back):
https://gerrit.ovirt.org/#/c/97491/ - Add pre-seed for ovirt-web-ui
Can you please check the issue as it seems both patches are changing the
same thing and the project seem to be broken since
https://gerrit.ovirt.org/#/c/97491/3
Latest failure:
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13171/
Logs:
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13171/arti...
errors from logs:
Engine:
2019-02-27 13:37:28,479-05 ERROR
[org.ovirt.engine.core.bll.UpdateVmCommand] (default task-1) [74283e25]
Transaction rolled-back for command
'org.ovirt.engine.core.bll.UpdateVmCommand'.
2019-02-27 13:37:28,483-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(default task-1) [74283e25] EVENT_ID: USER_FAILED_UPDATE_VM(58), Failed to
update VM vm2 (User: admin@inter
nal-authz).
2019-02-27 13:37:28,485-05 INFO
[org.ovirt.engine.core.bll.UpdateVmCommand] (default task-1) [74283e25]
Lock freed to object 'EngineLock:{exclusiveLocks='[vm2=VM_NAME]',
sharedLocks='[3500eb82-e5e2-4e24-b41c-ea
02d9f6adee=VM]'}'
2019-02-27 13:37:28,485-05 DEBUG
[org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
(default task-1) [74283e25] method: runAction, params: [UpdateVm,
VmManagementParametersBase:{commandId='340
59769-05b9-429e-8356-f6b9b9953f55', user='admin', commandType='UpdateVm',
vmId='3500eb82-e5e2-4e24-b41c-ea02d9f6adee'}], timeElapsed: 6618ms
2019-02-27 13:37:28,486-05 ERROR
[org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default
task-1) [] Operation Failed: []
2019-02-27 13:37:28,579-05 DEBUG
[org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
(EE-ManagedThreadFactory-engineScheduled-Thread-85) [] method: get, params:
[e29c0ba1-464c-4eb4-a8f2-c6933d9
9969a], timeElapsed: 3ms
vdsm:
2019-02-27 13:37:21,987-0500 INFO (jsonrpc/1) [vdsm.api] FINISH lease_info
error=No such lease 3500eb82-e5e2-4e24-b41c-ea02d9f6adee
from=::ffff:192.168.201.4,43920,
flow_id=117dec74-ad59-4b12-8148-b2c130337c10,
task_id=9c297d41-0aa7-4c74-b268-b710e666bc6c (api:52)
2019-02-27 13:37:21,988-0500 ERROR (jsonrpc/1) [storage.TaskManager.Task]
(Task='9c297d41-0aa7-4c74-b268-b710e666bc6c') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882,
in _run
return fn(*args, **kargs)
File "<string>", line 2, in lease_info
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3702,
in lease_info
info = dom.lease_info(lease.lease_id)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 674, in
lease_info
return vol.lookup(lease_id)
File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 553,
in lookup
raise NoSuchLease(lease_id)
NoSuchLease: No such lease 3500eb82-e5e2-4e24-b41c-ea02d9f6adee
2019-02-27 13:37:21,988-0500 INFO (jsonrpc/1) [storage.TaskManager.Task]
(Task='9c297d41-0aa7-4c74-b268-b710e666bc6c') aborting: Task is aborted:
u'No such lease 3500eb82-e5e2-4e24-b41c-ea02d9f6adee' - code 100
(task:1181)
2019-02-27 13:37:21,988-0500 ERROR (jsonrpc/1) [storage.Dispatcher] FINISH
lease_info error=No such lease 3500eb82-e5e2-4e24-b41c-ea02d9f6adee
(dispatcher:87)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line
74, in wrapper
result = ctask.prepare(func, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108,
in wrapper
return m(self, *a, **kw)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189,
in prepare
raise self.error
NoSuchLease: No such lease 3500eb82-e5e2-4e24-b41c-ea02d9f6adee
5 years, 8 months
Build failed in Jenkins:
system-sync_mirrors-fedora-base-fc29-x86_64 #370
by jenkins@jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/system-sync_mirrors-fedora-base-fc29-x86_64/...>
Changes:
[Galit Rosenthal] Nightly builds jobs for he-basic-ipv6_suite
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-fedora-base-fc29-x86_64/ws/>
No credentials specified
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune # timeout=10
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision fe1ffdd2713cf6fea1d37631b7b7cc4fba8d6524 (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f fe1ffdd2713cf6fea1d37631b7b7cc4fba8d6524 # timeout=10
Commit message: "Nightly builds jobs for he-basic-ipv6_suite"
> git rev-list --no-walk 27896049ffe1c8cc80346f7596e948086c275e99 # timeout=10
[system-sync_mirrors-fedora-base-fc29-x86_64] $ /bin/bash -xe /tmp/jenkins8158068874731989508.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf
+ MIRRORS_MP_BASE=/var/www/html/repos
+ MIRRORS_HTTP_BASE=http://mirrors.phx.ovirt.org/repos
+ MIRRORS_CACHE=/home/jenkins/mirrors_cache
+ MAX_LOCK_ATTEMPTS=120
+ LOCK_WAIT_INTERVAL=5
+ LOCK_BASE=/home/jenkins
+ OLD_MD_TO_KEEP=100
+ HTTP_SELINUX_TYPE=httpd_sys_content_t
+ HTTP_FILE_MODE=644
+ main resync_yum_mirror fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf
+ local command=resync_yum_mirror
+ command_args=("${@:2}")
+ local command_args
+ cmd_resync_yum_mirror fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf
+ local repo_name=fedora-base-fc29
+ local repo_archs=x86_64
+ local reposync_conf=jenkins/data/mirrors-reposync.conf
+ local sync_needed
+ mkdir -p /home/jenkins/mirrors_cache
+ verify_repo_fs fedora-base-fc29 yum
+ local repo_name=fedora-base-fc29
+ local repo_type=yum
+ sudo install -o jenkins -d /var/www/html/repos/yum /var/www/html/repos/yum/fedora-base-fc29 /var/www/html/repos/yum/fedora-base-fc29/base
+ check_yum_sync_needed fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf sync_needed
+ local repo_name=fedora-base-fc29
+ local repo_archs=x86_64
+ local reposync_conf=jenkins/data/mirrors-reposync.conf
+ local p_sync_needed=sync_needed
+ local reposync_out
+ echo 'Checking if mirror needs a resync'
Checking if mirror needs a resync
+ rm -rf /home/jenkins/mirrors_cache/fedora-base-fc29
++ IFS=,
++ echo x86_64
+ for arch in '$(IFS=,; echo $repo_archs)'
++ run_reposync fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf --urls --quiet
++ local repo_name=fedora-base-fc29
++ local repo_arch=x86_64
++ local reposync_conf=jenkins/data/mirrors-reposync.conf
++ extra_args=("${@:4}")
++ local extra_args
++ reposync --config=jenkins/data/mirrors-reposync.conf --repoid=fedora-base-fc29 --arch=x86_64 --cachedir=/home/jenkins/mirrors_cache --download_path=/var/www/html/repos/yum/fedora-base-fc29/base --norepopath --newest-only --urls --quiet
Traceback (most recent call last):
File "/usr/bin/reposync", line 343, in <module>
main()
File "/usr/bin/reposync", line 175, in main
my.doRepoSetup()
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
return self._getRepos(thisrepo, True)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
self._repos.doSetup(thisrepo)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
dl = repo._async and repo._commonLoadRepoXML(repo)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1479, in _commonLoadRepoXML
result = self._getFileRepoXML(local, text)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1256, in _getFileRepoXML
size=102400) # setting max size as 100K
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1022, in _getFile
result = self.grab.urlgrab(misc.to_utf8(relative), local,
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 700, in <lambda>
grab = property(lambda self: self._getgrab())
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 695, in _getgrab
self._setupGrab()
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 632, in _setupGrab
urls = self.urls
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 878, in <lambda>
urls = property(fget=lambda self: self._geturls(),
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 875, in _geturls
self._baseurlSetup()
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 821, in _baseurlSetup
mirrorurls.extend(list(self.metalink_data.urls()))
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 918, in <lambda>
metalink_data = property(fget=lambda self: self._getMetalink(),
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 914, in _getMetalink
self._metalink = metalink.MetaLinkRepoMD(self.metalink_filename)
File "/usr/lib/python2.7/site-packages/yum/metalink.py", line 185, in __init__
raise MetaLinkRepoErrorParseFail, "File %s does not exist" %filename
yum.metalink.MetaLinkRepoErrorParseFail: File /home/jenkins/mirrors_cache/fedora-base-fc29/metalink.xml does not exist
+ reposync_out='Could not parse metalink https://mirrors.fedoraproject.org/metalink?repo=fedora-29&arch=x86_64 error was
File /home/jenkins/mirrors_cache/fedora-base-fc29/metalink.xml.tmp is not XML'
Build step 'Execute shell' marked build as failure
5 years, 8 months
Build failed in Jenkins:
system-sync_mirrors-centos-kvm-common-el7-x86_64 #2269
by jenkins@jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-kvm-common-el7-x8...>
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-kvm-common-el7-x8...>
No credentials specified
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/changes/59/95959/1:test --prune # timeout=10
> git rev-parse origin/test^{commit} # timeout=10
> git rev-parse test^{commit} # timeout=10
Checking out Revision 05b40dfb4ec43a82530e8c471395d1858e5c59e1 (test)
> git config core.sparsecheckout # timeout=10
> git checkout -f 05b40dfb4ec43a82530e8c471395d1858e5c59e1 # timeout=10
Commit message: "mirror-reposync: remove gluster-3.10 mirror"
> git rev-list --no-walk 05b40dfb4ec43a82530e8c471395d1858e5c59e1 # timeout=10
[system-sync_mirrors-centos-kvm-common-el7-x86_64] $ /bin/bash -xe /tmp/jenkins354109812899683677.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-kvm-common-el7 x86_64 jenkins/data/mirrors-reposync.conf
Checking if mirror needs a resync
Traceback (most recent call last):
File "/usr/bin/reposync", line 343, in <module>
main()
File "/usr/bin/reposync", line 175, in main
my.doRepoSetup()
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
return self._getRepos(thisrepo, True)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
self._repos.doSetup(thisrepo)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
dl = repo._async and repo._commonLoadRepoXML(repo)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1479, in _commonLoadRepoXML
result = self._getFileRepoXML(local, text)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1256, in _getFileRepoXML
size=102400) # setting max size as 100K
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1022, in _getFile
result = self.grab.urlgrab(misc.to_utf8(relative), local,
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 700, in <lambda>
grab = property(lambda self: self._getgrab())
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 695, in _getgrab
self._setupGrab()
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 632, in _setupGrab
urls = self.urls
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 878, in <lambda>
urls = property(fget=lambda self: self._geturls(),
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 875, in _geturls
self._baseurlSetup()
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 821, in _baseurlSetup
mirrorurls.extend(list(self.metalink_data.urls()))
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 918, in <lambda>
metalink_data = property(fget=lambda self: self._getMetalink(),
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 897, in _getMetalink
raise Errors.RepoError, msg
yum.Errors.RepoError: Cannot retrieve metalink for repository: fedora-base-fc29. Please verify its path and try again
Build step 'Execute shell' marked build as failure
5 years, 8 months