- Infra - oVirt List Archives

oVirt infra weekly meeting notes 08.08.2018
by Evgheni Dereveanchin 11 Aug '18

11 Aug '18

Please find the topics of this week's infra meeting below: Note: meeting now happens at a different time: 11 AM GMT on Wednesdays PHX: - Storage migration: OVIRT-1490 - NFS migrated successfully - storage01 shut down, ready for rebuild - ovirt-srv06 rebuilt with LVM and containers deployed OVIRT-2409 - Old openshift instance shut down - mirror monitoring needs to be moved to new instance OVIRT-1801 - Monitoring sent some false-positives - root cause TBD OVIRT-2406 -- Regards, Evgheni Dereveanchin

2 1

[oVirt Jenkins] ovirt-system-tests_compat-4.2-suite-master - Build # 61 - Failure!
by jenkins＠jenkins.phx.ovirt.org 11 Aug '18

11 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.2-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.2-suite-master/61/ Build Number: 61 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #61 [Gal Ben Haim] HE: Added Local Maintenance Tests ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.verify_vm1_exported_42 Error Message: 'NoneType' object has no attribute 'get' -------------------- >> begin captured logging << -------------------- ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7ff8c80ce410> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/004_basic_sanity.py", line 536, in <lambda> vm_sd_service.get().status == types.VmStatus.DOWN AttributeError: 'NoneType' object has no attribute 'get' --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/004_basic_sanity.py", line 535, in verify_vm1_exported lambda: File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 282, in assert_true_within_short assert_equals_within_short(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 266, in assert_equals_within_short func, value, SHORT_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/004_basic_sanity.py", line 536, in <lambda> vm_sd_service.get().status == types.VmStatus.DOWN '\'NoneType\' object has no attribute \'get\'\n-------------------- >> begin captured logging << --------------------\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7ff8c80ce410>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/004_basic_sanity.py", line 536, in <lambda>\n vm_sd_service.get().status == types.VmStatus.DOWN\nAttributeError: \'NoneType\' object has no attribute \'get\'\n--------------------- >> end captured logging << ---------------------'

1 1

oVirt infra daily report - unstable production jobs - 549
by jenkins＠jenkins.phx.ovirt.org 11 Aug '18

11 Aug '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/549//artifact/exported-a… Cheers, Jenkins

1 0

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #1834
by jenkins＠jenkins.phx.ovirt.org 11 Aug '18

11 Aug '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 33fda589b17441ff0a3458558676707884702007 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 33fda589b17441ff0a3458558676707884702007 Commit message: "ovirt-wgt: exclude fc28 on 4.2" > git rev-list --no-walk 33fda589b17441ff0a3458558676707884702007 # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins1563323248376698292.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7' Build step 'Execute shell' marked build as failure

1 1

[CQ]: 93610, 2 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 11 Aug '18

11 Aug '18

A system test invoked by the "ovirt-master" change queue including change 93610,2 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 93508,1 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 93508,1 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/93610/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/93508/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/9039/

1 0

Build failed in Jenkins: system-sync_mirrors-epel-el6-x86_64 #1755
by jenkins＠jenkins.phx.ovirt.org 11 Aug '18

11 Aug '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/1755/displ…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 33fda589b17441ff0a3458558676707884702007 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 33fda589b17441ff0a3458558676707884702007 Commit message: "ovirt-wgt: exclude fc28 on 4.2" > git rev-list --no-walk 33fda589b17441ff0a3458558676707884702007 # timeout=10 [system-sync_mirrors-epel-el6-x86_64] $ /bin/bash -xe /tmp/jenkins1996224519354460097.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror epel-el6 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1474, in _commonLoadRepoXML if self._latestRepoXML(local): File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1443, in _latestRepoXML oxml = self._saveOldRepoXML(local) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1300, in _saveOldRepoXML shutil.copy2(local, old_local) File "/usr/lib64/python2.7/shutil.py", line 131, in copy2 copystat(src, dst) File "/usr/lib64/python2.7/shutil.py", line 98, in copystat os.utime(dst, (st.st_atime, st.st_mtime)) OSError: [Errno 2] No such file or directory: '/home/jenkins/mirrors_cache/centos-extras-el7/repomd.xml.old.tmp' Build step 'Execute shell' marked build as failure

1 1

oVirt infra daily report - unstable production jobs - 548
by jenkins＠jenkins.phx.ovirt.org 10 Aug '18

10 Aug '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/548//artifact/exported-a… Cheers, Jenkins

1 0

[CQ]: 93634,2 (cockpit-ovirt) failed "ovirt-master" system tests
by oVirt Jenkins 10 Aug '18

10 Aug '18

Change 93634,2 (cockpit-ovirt) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/93634/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/9033/

1 0

[CQ]: 93638, 2 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 10 Aug '18

10 Aug '18

A system test invoked by the "ovirt-master" change queue including change 93638,2 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 93508,1 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 93508,1 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/93638/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/93508/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/9028/

1 0

[ OST Failure Report ] [ oVirt 4.2 (imgbased) ] [ 09-08-2018 ] [ 004_basic_sanity.check_snapshot_with_memory ]
by Dafna Ron 10 Aug '18

10 Aug '18

Hi, We have a failure in 4.2 which I think may be related to the patch itself. Jira opened with additional info: https://ovirt-jira.atlassian.net/browse/OVIRT-2418 *Link and headline of suspected patches: * *We failed patch https://gerrit.ovirt.org/#/c/93545/ <https://gerrit.ovirt.org/#/c/93545/> - core: remove lvs after a failed upgrade on ovirt-4.2Link to Job:* * https://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2819/ <https://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2819/>Link to all logs:* * https://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2819/artifact/b… <https://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2819/>(Relevant) error snippet from the log: <error>* Error Message Fault reason is "Operation Failed". Fault detail is "[Cannot run VM. Low disk space on Storage Domain iscsi.]". HTTP response code is 409. Stacktrace Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/workspace/ovirt-4.2_change-queue-tester/ovirt-system-tests/basic-suite-4.2/test-scenarios/004_basic_sanity.py", line 589, in check_snapshot_with_memory vm_service.start() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 30074, in start return self._internal_action(action, 'start', None, headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 299, in _internal_action return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 296, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 134, in _check_fault self._raise_error(response, body.fault) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error Error: Fault reason is "Operation Failed". Fault detail is "[Cannot run VM. Low disk space on Storage Domain iscsi.]". HTTP response code is 409. *</error>*

3 2