discussing the future of the upgrade suite in ost
by Dafna Ron
Hi All,
I was reviewing the upgrade suites in ost and there are some issues that I
am seeing in the suite tests-scenarios which I want to discuss and decide
the future of.
At it current state, I think we should remove the upgrade suite or most of
the post test-scenarios as it is not testing what it should.
The tests currently only test engine upgrade and basic sanity after the
upgrade.
This is problematic in a few ways:
1. upgrade should test the upgrade of rhv and not just a clean engine
upgrade (i.e host, storage, vm).
2. as we have limited resources I do not think that the upgrade suite
should be longer then the basic suite (and as we are currently running the
basic suite after the upgrade it is longer)
That brings me to question what should be essential to test in upgrade in
the CI?
I would also need someone in dev to volunteer and take ownership of the
testing scenarios for upgrade - is there anyone that can help?
Thanks,
Dafna
6 years, 2 months
Build failed in Jenkins:
system-sync_mirrors-centos-updates-el7-x86_64 #2027
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...>
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...>
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 92c04a259244f3206b0c9a92617876fb2e23ec3d (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 92c04a259244f3206b0c9a92617876fb2e23ec3d
Commit message: "standard-pipelines: add kubevirt/common-templates"
> git rev-list --no-walk 92c04a259244f3206b0c9a92617876fb2e23ec3d # timeout=10
[system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins2726344544018812070.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf
Checking if mirror needs a resync
Traceback (most recent call last):
File "/usr/bin/reposync", line 343, in <module>
main()
File "/usr/bin/reposync", line 175, in main
my.doRepoSetup()
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
return self._getRepos(thisrepo, True)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
self._repos.doSetup(thisrepo)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
dl = repo._async and repo._commonLoadRepoXML(repo)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML
local = self.cachedir + '/repomd.xml'
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda>
cachedir = property(lambda self: self._dirGetAttr('cachedir'))
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr
self.dirSetup()
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup
self._dirSetupMkdir_p(dir)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p
raise Errors.RepoError, msg
yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7/gen error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7/gen'
Build step 'Execute shell' marked build as failure
6 years, 2 months
[JENKINS] Failed to setup proejct
kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org
Failed to run project_setup.sh for:
#2176 kubevirt [check-patch].
It probably means that docker_cleanup.py failed.
This step doesn't fail the job, but we do collect
data about such failures to find the root cause.
Infra owner, ensure that we're not running out of
disk space on ovirt-srv04.phx.ovirt.org-container-4.
6 years, 2 months
[oVirt Jenkins] ovirt-system-tests_he-basic-ansible-suite-master -
Build # 708 - Failure!
by jenkins＠jenkins.phx.ovirt.org
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-ma...
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-ma...
Build Number: 708
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #708
[Eitan Raviv] networking: connect NFS to engine over IPv6
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: 002_bootstrap.install_cockpit_ovirt
Error Message:
_install_cockpit_ovirt_on_host(): failed to install cockpit-ovirt-dashboard on host <ovirtlago.virt.HostVM object at 0x7f565ad34810>
-------------------- >> begin captured logging << --------------------
lago.ssh: DEBUG: start task:11772f94-8884-45cf-a992-ffea1dc9a633:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: start task:4188d33b-fd14-4ab7-b5ce-a53c1b970597:Get ssh client for lago-he-basic-ansible-suite-master-host-1:
lago.ssh: DEBUG: end task:4188d33b-fd14-4ab7-b5ce-a53c1b970597:Get ssh client for lago-he-basic-ansible-suite-master-host-1:
lago.ssh: DEBUG: end task:11772f94-8884-45cf-a992-ffea1dc9a633:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running 1ce699c0 on lago-he-basic-ansible-suite-master-host-0: yum -y install cockpit-ovirt-dashboard
lago.ssh: DEBUG: Running 1ceadc4c on lago-he-basic-ansible-suite-master-host-1: yum -y install cockpit-ovirt-dashboard
lago.ssh: DEBUG: Command 1ceadc4c on lago-he-basic-ansible-suite-master-host-1 returned with 0
lago.ssh: DEBUG: Command 1ceadc4c on lago-he-basic-ansible-suite-master-host-1 output:
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id,
: search-disabled-repos, subscription-manager, vdsmupgrade
This system is not registered with an entitlement server. You can use subscription-manager to register.
Determining fastest mirrors
* base: repos.lax.quadranet.com
* extras: mirror.fileplanet.com
* updates: mirror.scalabledns.com
Package cockpit-ovirt-dashboard-0.11.999-0.0.master.20180927.git6281078.el7.noarch already installed and latest version
Nothing to do
Uploading Enabled Repositories Report
Loaded plugins: fastestmirror, product-id, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
lago.ssh: DEBUG: Command 1ceadc4c on lago-he-basic-ansible-suite-master-host-1 errors:
Cannot upload enabled repos report, is this client registered?
lago.ssh: DEBUG: Command 1ce699c0 on lago-he-basic-ansible-suite-master-host-0 returned with 1
lago.ssh: DEBUG: Command 1ce699c0 on lago-he-basic-ansible-suite-master-host-0 output:
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id,
: search-disabled-repos, subscription-manager, vdsmupgrade
This system is not registered with an entitlement server. You can use subscription-manager to register.
Determining fastest mirrors
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=stock error was
12: Timeout on http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=stock: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
Uploading Enabled Repositories Report
Loaded plugins: fastestmirror, product-id, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
lago.ssh: DEBUG: Command 1ce699c0 on lago-he-basic-ansible-suite-master-host-0 errors:
BDB2053 Freeing read locks for locker 0x101: 21385/140097655068736
BDB2053 Freeing read locks for locker 0x103: 21385/140097655068736
BDB2053 Freeing read locks for locker 0x104: 21385/140097655068736
BDB2053 Freeing read locks for locker 0x105: 21385/140097655068736
One of the configured repositories failed (Unknown),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:
1. Contact the upstream for the repository and get them to fix the problem.
2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work).
3. Run the command with the repository temporarily disabled
yum --disablerepo=<repoid> ...
4. Disable the repository permanently, so yum won't use it by default. Yum
will then just ignore the repository until you permanently enable it
again or use --enablerepo for temporary usage:
yum-config-manager --disable <repoid>
or
subscription-manager repos --disable=<repoid>
5. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise:
yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true
Cannot find a valid baseurl for repo: base/7/x86_64
Cannot upload enabled repos report, is this client registered?
lago.utils: ERROR: Error while running thread
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue
queue.put({'return': func()})
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/002_bootstrap.py", line 414, in _install_cockpit_ovirt_on_host
nt.assert_equals(ret.code, 0, '_install_cockpit_ovirt_on_host(): failed to install cockpit-ovirt-dashboard on host %s' % host)
File "/usr/lib64/python2.7/unittest/case.py", line 553, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/lib64/python2.7/unittest/case.py", line 546, in _baseAssertEqual
raise self.failureException(msg)
AssertionError: _install_cockpit_ovirt_on_host(): failed to install cockpit-ovirt-dashboard on host <ovirtlago.virt.HostVM object at 0x7f565ad34810>
--------------------- >> end captured logging << ---------------------
Stack Trace:
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/002_bootstrap.py", line 421, in install_cockpit_ovirt
nt.assert_true(all(vt.join_all()), 'not all threads finished: %s' % vt)
File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue
queue.put({'return': func()})
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/002_bootstrap.py", line 414, in _install_cockpit_ovirt_on_host
nt.assert_equals(ret.code, 0, '_install_cockpit_ovirt_on_host(): failed to install cockpit-ovirt-dashboard on host %s' % host)
File "/usr/lib64/python2.7/unittest/case.py", line 553, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/lib64/python2.7/unittest/case.py", line 546, in _baseAssertEqual
raise self.failureException(msg)
'_install_cockpit_ovirt_on_host(): failed to install cockpit-ovirt-dashboard on host <ovirtlago.virt.HostVM object at 0x7f565ad34810>\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:11772f94-8884-45cf-a992-ffea1dc9a633:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: start task:4188d33b-fd14-4ab7-b5ce-a53c1b970597:Get ssh client for lago-he-basic-ansible-suite-master-host-1:\nlago.ssh: DEBUG: end task:4188d33b-fd14-4ab7-b5ce-a53c1b970597:Get ssh client for lago-he-basic-ansible-suite-master-host-1:\nlago.ssh: DEBUG: end task:11772f94-8884-45cf-a992-ffea1dc9a633:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 1ce699c0 on lago-he-basic-ansible-suite-master-host-0: yum -y install cockpit-ovirt-dashboard\nlago.ssh: DEBUG: Running 1ceadc4c on lago-he-basic-ansible-suite-master-host-1: yum -y install cockpit-ovirt-dashboard\nlago.ssh: DEBUG: Command 1ceadc4c on lago-he-basic-ansible-suite-master-host-1 returned with 0\nlago.ssh: DEBUG: Command 1ceadc4c on lago-he-basic-ansible-suite-master-host-1 output:\n Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id,\n : search-disabled-repos, subscription-manager, vdsmupgrade\nThis system is not registered with an entitlement server. You can use subscription-manager to register.\nDetermining fastest mirrors\n * base: repos.lax.quadranet.com\n * extras: mirror.fileplanet.com\n * updates: mirror.scalabledns.com\nPackage cockpit-ovirt-dashboard-0.11.999-0.0.master.20180927.git6281078.el7.noarch already installed and latest version\nNothing to do\nUploading Enabled Repositories Report\nLoaded plugins: fastestmirror, product-id, subscription-manager\nThis system is not registered with an entitlement server. You can use subscription-manager to register.\n\nlago.ssh: DEBUG: Command 1ceadc4c on lago-he-basic-ansible-suite-master-host-1 errors:\n Cannot upload enabled repos report, is this client registered?\n\nlago.ssh: DEBUG: Command 1ce699c0 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 1ce699c0 on lago-he-basic-ansible-suite-master-host-0 output:\n Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id,\n : search-disabled-repos, subscription-manager, vdsmupgrade\nThis system is not registered with an entitlement server. You can use subscription-manager to register.\nDetermining fastest mirrors\nCould not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=stock error was\n12: Timeout on http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=stock: (28, \'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds\')\nUploading Enabled Repositories Report\nLoaded plugins: fastestmirror, product-id, subscription-manager\nThis system is not registered with an entitlement server. You can use subscription-manager to register.\n\nlago.ssh: DEBUG: Command 1ce699c0 on lago-he-basic-ansible-suite-master-host-0 errors:\n BDB2053 Freeing read locks for locker 0x101: 21385/140097655068736\nBDB2053 Freeing read locks for locker 0x103: 21385/140097655068736\nBDB2053 Freeing read locks for locker 0x104: 21385/140097655068736\nBDB2053 Freeing read locks for locker 0x105: 21385/140097655068736\n\n\n One of the configured repositories failed (Unknown),\n and yum doesn\'t have enough cached data to continue. At this point the only\n safe thing yum can do is fail. There are a few ways to work "fix" this:\n\n 1. Contact the upstream for the repository and get them to fix the problem.\n\n 2. Reconfigure the baseurl/etc. for the repository, to point to a working\n upstream. This is most often useful if you are using a newer\n distribution release than is supported by the repository (and the\n packages for the previous distribution release still work).\n\n 3. Run the command with the repository temporarily disabled\n yum --disablerepo=<repoid> ...\n\n 4. Disable the repository permanently, so yum won\'t use it by default. Yum\n will then just ignore the repository until you permanently enable it\n again or use --enablerepo for temporary usage:\n\n yum-config-manager --disable <repoid>\n or\n subscription-manager repos --disable=<repoid>\n\n 5. Configure the failing repository to be skipped, if it is unavailable.\n Note that yum will try to contact the repo. when it runs most commands,\n so will have to try and fail each time (and thus. yum will be be much\n slower). If it is a very temporary problem though, this is often a nice\n compromise:\n\n yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true\n\nCannot find a valid baseurl for repo: base/7/x86_64\nCannot upload enabled repos report, is this client registered?\n\nlago.utils: ERROR: Error while running thread\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue\n queue.put({\'return\': func()})\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/002_bootstrap.py", line 414, in _install_cockpit_ovirt_on_host\n nt.assert_equals(ret.code, 0, \'_install_cockpit_ovirt_on_host(): failed to install cockpit-ovirt-dashboard on host %s\' % host)\n File "/usr/lib64/python2.7/unittest/case.py", line 553, in assertEqual\n assertion_func(first, second, msg=msg)\n File "/usr/lib64/python2.7/unittest/case.py", line 546, in _baseAssertEqual\n raise self.failureException(msg)\nAssertionError: _install_cockpit_ovirt_on_host(): failed to install cockpit-ovirt-dashboard on host <ovirtlago.virt.HostVM object at 0x7f565ad34810>\n--------------------- >> end captured logging << ---------------------'
6 years, 2 months
