- Infra - oVirt List Archives

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-4.2 - Build # 642 - Failure!
by jenkins＠jenkins.phx.ovirt.org 11 Nov '18

11 Nov '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/642/ Build Number: 642 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #642 [Eitan Raviv] networking: sync networks across cluster ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.add_hosts Error Message: 143 -------------------- >> begin captured logging << -------------------- cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 229, in add_hosts add_hosts_4(prefix) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 273, in add_hosts_4 testlib.assert_true_within(_host_is_up_4, timeout=15*60) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 278, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 254, in _host_is_up_4 host_obj = host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37093, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 54, in wait response = self._connection.wait(self._context) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 496, in wait return self.__wait(context, failed_auth) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 532, in __wait self._multi.select(1.0) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'

1 4

Build failed in Jenkins: system-sync_mirrors-epel-el6-x86_64 #2036
by jenkins＠jenkins.phx.ovirt.org 11 Nov '18

11 Nov '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/2036/displ…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision e4f74212268b5757494d69d283eccc20fbe92a97 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f e4f74212268b5757494d69d283eccc20fbe92a97 Commit message: "Time-based trigger openshift-on-ovirt suite" > git rev-list --no-walk e4f74212268b5757494d69d283eccc20fbe92a97 # timeout=10 [system-sync_mirrors-epel-el6-x86_64] $ /bin/bash -xe /tmp/jenkins4477561812353850902.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror epel-el6 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7' Build step 'Execute shell' marked build as failure

1 1

Build failed in Jenkins: ovirt_4.1_publish-rpms_nightly #712
by jenkins＠jenkins.phx.ovirt.org 11 Nov '18

11 Nov '18

See <http://jenkins.ovirt.org/job/ovirt_4.1_publish-rpms_nightly/712/display/red…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building on master in workspace <http://jenkins.ovirt.org/job/ovirt_4.1_publish-rpms_nightly/ws/> [WS-CLEANUP] Deleting project workspace... [workspace] $ /bin/bash -xe /tmp/jenkins1869602453035378154.sh + rm -rf <http://jenkins.ovirt.org/job/ovirt_4.1_publish-rpms_nightly/ws/artifacts> + mkdir <http://jenkins.ovirt.org/job/ovirt_4.1_publish-rpms_nightly/ws/artifacts> Copied 7 artifacts from "otopi_4.1_build-artifacts-el7-x86_64" build number 20 Copied 5 artifacts from "ovirt-vmconsole_4.1_build-artifacts-el7-x86_64" build number 18 Copied 4 artifacts from "ovirt-log-collector_4.1_build-artifacts-el7-x86_64" build number 210 Copied 3 artifacts from "ovirt-engine-cli_3.6_build-artifacts-el7-x86_64" build number 27 Copied 3 artifacts from "ovirt-engine-extension-aaa-jdbc_4.1_build-artifacts-el7-x86_64" build number 25 Copied 4 artifacts from "ovirt-engine-extension-aaa-ldap_4.1_build-artifacts-el7-x86_64" build number 52 Copied 3 artifacts from "ovirt-engine-extension-aaa-misc_4.1_build-artifacts-el7-x86_64" build number 7 Copied 3 artifacts from "ovirt-engine-extension-logger-log4j_4.1_build-artifacts-el7-x86_64" build number 12 Copied 4 artifacts from "vdsm-jsonrpc-java_4.1_build-artifacts-el7-x86_64" build number 34 Copied 4 artifacts from "python-ovirt-engine-sdk4_4.1_build-artifacts-el7-x86_64" build number 114 Copied 4 artifacts from "python-ovirt-engine-sdk4_4.1_build-artifacts-el7-ppc64le" build number 86 Copied 4 artifacts from "ovirt-engine-sdk-java_4.1_build-artifacts-el7-x86_64" build number 11 Copied 3 artifacts from "java-ovirt-engine-sdk4_4.1_build-artifacts-el7-x86_64" build number 65 Copied 3 artifacts from "ovirt-scheduler-proxy_4.1_build-artifacts-el7-x86_64" build number 11 Copied 5 artifacts from "ovirt-optimizer_4.1_build-artifacts-el7-x86_64" build number 11 Copied 3 artifacts from "ovirt-engine-dashboard_4.1_build-artifacts-el7-x86_64" build number 48 Copied 5 artifacts from "ioprocess_4.1_build-artifacts-el7-x86_64" build number 1 Copied 5 artifacts from "ioprocess_4.1_build-artifacts-el7-ppc64le" build number 1 Copied 51 artifacts from "vdsm_4.1_build-artifacts-el7-x86_64" build number 483 Copied 51 artifacts from "vdsm_4.1_build-artifacts-el7-ppc64le" build number 481 Copied 3 artifacts from "ovirt-hosted-engine-ha_4.1_build-artifacts-el7-x86_64" build number 93 Copied 3 artifacts from "ovirt-hosted-engine-setup_4.1_build-artifacts-el7-x86_64" build number 95 Copied 3 artifacts from "ovirt-engine-sdk_4.1_build-artifacts-el7-x86_64" build number 2 Copied 22 artifacts from "ovirt-engine_4.1_build-artifacts-el7-x86_64" build number 1124 Copied 3 artifacts from "mom_4.1_build-artifacts-el7-x86_64" build number 17 Copied 3 artifacts from "ovirt-wgt_4.1_build-artifacts-el7-x86_64" build number 1 Copied 4 artifacts from "cockpit-ovirt_4.1_build-artifacts-el7-x86_64" build number 126 Copied 9 artifacts from "ovirt-engine-sdk-ruby_4.1_build-artifacts-el7-x86_64" build number 159 Copied 9 artifacts from "ovirt-engine-sdk-ruby_4.1_build-artifacts-el7-ppc64le" build number 163 Copied 3 artifacts from "ovirt-guest-agent_4.1_build-artifacts-el6-x86_64" build number 8 Copied 7 artifacts from "ovirt-guest-agent_4.1_build-artifacts-el7-x86_64" build number 10 ERROR: Unable to find a build for artifact copy from: ovirt-provider-ovn_4.1_build-artifacts-el7-x86_64 SSH: Current build result is [FAILURE], not going to run.

1 24

oVirt infra daily report - unstable production jobs - 640
by jenkins＠jenkins.phx.ovirt.org 11 Nov '18

11 Nov '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/640//artifact/exported-a… Cheers, Jenkins

1 0

[ OST Failure Report ] [ oVirt Master (ovirt-engine) ] [ 08-11-2018 ] [ verify_suspend_resume_vm0 ]
by Dafna Ron 10 Nov '18

10 Nov '18

Hi, We have a regression in ovirt-engine which is causing a failure in test verify_suspend_resume_vm0 in basic suite. Since the regression was introduced along side a second regression CQ was unable to point on the faulty patch, only on the first failed patch since the new regression. Dominik debugged the issue and found the problematic change: https://gerrit.ovirt.org/#/c/95222/ - engine : Updating template of VM Pool leaves tasks stuck after VMs shutdown Jira: https://ovirt-jira.atlassian.net/browse/OVIRT-2571 error from engine log: 2018-11-08 04:26:42,081-05 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-1) [] Operation Failed: [Cannot remove VM-Pool. VM is being updated.] Thanks, Dafna

2 1

[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-master - Build # 545 - Failure!
by jenkins＠jenkins.phx.ovirt.org 10 Nov '18

10 Nov '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/545/ Build Number: 545 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #545 [Eitan Raviv] networking: sync networks across cluster ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 012_local_maintenance_sdk.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... root: INFO: * Performing Deactivation... root: INFO: * Performing Activation... root: INFO: * Waiting For System Stability... cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/012_local_maintenance_sdk.py", line 79, in local_maintenance time.sleep(wait_value) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nroot: INFO: * Performing Deactivation...\nroot: INFO: * Performing Activation...\nroot: INFO: * Waiting For System Stability...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'

1 3

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #2120
by jenkins＠jenkins.phx.ovirt.org 10 Nov '18

10 Nov '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision e4f74212268b5757494d69d283eccc20fbe92a97 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f e4f74212268b5757494d69d283eccc20fbe92a97 Commit message: "Time-based trigger openshift-on-ovirt suite" > git rev-list --no-walk e4f74212268b5757494d69d283eccc20fbe92a97 # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins5491587166523542718.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/epel-el6 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/epel-el6' Build step 'Execute shell' marked build as failure

1 1

[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-4.2 - Build # 477 - Failure!
by jenkins＠jenkins.phx.ovirt.org 10 Nov '18

10 Nov '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/477/ Build Number: 477 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #477 [Eitan Raviv] networking: sync networks across cluster ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 012_local_maintenance_sdk.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... root: INFO: * Performing Deactivation... root: INFO: * Performing Activation... root: INFO: * Waiting For System Stability... cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/012_local_maintenance_sdk.py", line 79, in local_maintenance time.sleep(wait_value) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nroot: INFO: * Performing Deactivation...\nroot: INFO: * Performing Activation...\nroot: INFO: * Waiting For System Stability...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'

1 4

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build # 863 - Failure!
by jenkins＠jenkins.phx.ovirt.org 10 Nov '18

10 Nov '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/863/ Build Number: 863 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #863 [Eitan Raviv] networking: sync networks across cluster ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.add_hosts Error Message: Failed to read response: [(<pycurl.Curl object at 0x7efcf2399050>, 7, 'TCP connection reset by peer')] -------------------- >> begin captured logging << -------------------- ovirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7efcf3822320> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4 host_obj = host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 38639, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 54, in wait response = self._connection.wait(self._context) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 497, in wait return self.__wait(context, failed_auth) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 511, in __wait raise Error("Failed to read response: {}".format(err_list)) Error: Failed to read response: [(<pycurl.Curl object at 0x7efcf2399050>, 7, 'TCP connection reset by peer')] --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 228, in add_hosts add_hosts_4(prefix) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 275, in add_hosts_4 testlib.assert_true_within(_host_is_up_4, timeout=15*60) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 278, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4 host_obj = host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 38639, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 54, in wait response = self._connection.wait(self._context) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 497, in wait return self.__wait(context, failed_auth) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 511, in __wait raise Error("Failed to read response: {}".format(err_list)) 'Failed to read response: [(<pycurl.Curl object at 0x7efcf2399050>, 7, \'TCP connection reset by peer\')]\n-------------------- >> begin captured logging << --------------------\novirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7efcf3822320>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4\n host_obj = host_service.get()\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 38639, in get\n return self._internal_get(headers, query, wait)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get\n return future.wait() if wait else future\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 54, in wait\n response = self._connection.wait(self._context)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 497, in wait\n return self.__wait(context, failed_auth)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 511, in __wait\n raise Error("Failed to read response: {}".format(err_list))\nError: Failed to read response: [(<pycurl.Curl object at 0x7efcf2399050>, 7, \'TCP connection reset by peer\')]\n--------------------- >> end captured logging << ---------------------'

1 1

[CQ]: 95336, 3 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 10 Nov '18

10 Nov '18

A system test invoked by the "ovirt-master" change queue including change 95336,3 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 95190,2 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95190,2 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/95336/3 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95190/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11110/

1 0

[oVirt Jenkins] ovirt-system-tests_he-basic-ansible-suite-master - Build # 798 - Failure!
by jenkins＠jenkins.phx.ovirt.org 10 Nov '18

10 Nov '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-mast… Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-mast… Build Number: 798 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #798 [Eitan Raviv] networking: sync networks across cluster ----------------- Failed Tests: ----------------- No tests ran.

1 1

oVirt infra daily report - unstable production jobs - 639
by jenkins＠jenkins.phx.ovirt.org 10 Nov '18

10 Nov '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/639//artifact/exported-a… Cheers, Jenkins

1 0

[CQ]: 95310, 7 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 09 Nov '18

09 Nov '18

A system test invoked by the "ovirt-master" change queue including change 95310,7 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 95190,2 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95190,2 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/95310/7 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95190/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11107/

1 0

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 09 Nov '18

09 Nov '18

Failed to run project_setup.sh for: #2460 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on ovirt-srv04.phx.ovirt.org-container-4.

1 0

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 09 Nov '18

09 Nov '18

Failed to run project_setup.sh for: #2460 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on ovirt-srv04.phx.ovirt.org-container-1.

1 0

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 09 Nov '18

09 Nov '18

Failed to run project_setup.sh for: #2460 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on ovirt-srv04.phx.ovirt.org-container-5.

1 0

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 09 Nov '18

09 Nov '18

Failed to run project_setup.sh for: #2460 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on ovirt-srv05.phx.ovirt.org-container-5.

1 0

[CQ]: 95283, 3 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 09 Nov '18

09 Nov '18

A system test invoked by the "ovirt-master" change queue including change 95283,3 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 95190,2 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95190,2 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/95283/3 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95190/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11103/

1 0

[CQ]: 95281, 4 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 09 Nov '18

09 Nov '18

A system test invoked by the "ovirt-master" change queue including change 95281,4 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 95190,2 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95190,2 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/95281/4 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95190/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11100/

1 0

[CQ]: 94980,11 (ovirt-provider-ovn) failed "ovirt-4.2" system tests
by oVirt Jenkins 09 Nov '18

09 Nov '18

Change 94980,11 (ovirt-provider-ovn) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/94980/11 For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3468/

2 1

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build # 861 - Failure!
by jenkins＠jenkins.phx.ovirt.org 09 Nov '18

09 Nov '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/861/ Build Number: 861 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #861 [Eitan Raviv] networking: sync networks across cluster ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.add_hosts Error Message: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML -------------------- >> begin captured logging << -------------------- ovirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7f28df143320> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4 host_obj = host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 38639, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 850, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 890, in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 228, in add_hosts add_hosts_4(prefix) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 275, in add_hosts_4 testlib.assert_true_within(_host_is_up_4, timeout=15*60) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 278, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4 host_obj = host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 38639, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 850, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 890, in _check_content_type raise Error(msg) 'The response content type \'text/html; charset=iso-8859-1\' isn\'t the expected XML\n-------------------- >> begin captured logging << --------------------\novirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7f28df143320>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4\n host_obj = host_service.get()\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 38639, in get\n return self._internal_get(headers, query, wait)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get\n return future.wait() if wait else future\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait\n return self._code(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback\n self._check_fault(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault\n body = self._internal_read_body(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body\n self._connection.check_xml_content_type(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 850, in check_xml_content_type\n response.headers\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 890, in _check_content_type\n raise Error(msg)\nError: The response content type \'text/html; charset=iso-8859-1\' isn\'t the expected XML\n--------------------- >> end captured logging << ---------------------'

1 1

[JIRA] (OVIRT-2572) ovirt-provider-ovn fails in 4.2 due to lago's repolock
by Dafna Ron (oVirt JIRA) 09 Nov '18

09 Nov '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2572?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2572: ---------------------------------- I am unable to ssh to the host so cannot clean the lock dron@dron ~]$ ssh ovirt-srv21.phx.ovirt.org ssh: connect to host ovirt-srv21.phx.ovirt.org port 22: Connection timed out [dron@dron ~]$ > ovirt-provider-ovn fails in 4.2 due to lago's repolock > ------------------------------------------------------ > > Key: OVIRT-2572 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2572 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_infra > > build that failed: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3468/ > basic suite was running on ovirt-srv21.phx.ovirt.org > {noformat} > 10:33:23 [basic-suite.el7.x86_64] @ Create prefix internal repo: ERROR (in 0:03:00) > 10:33:23 [basic-suite.el7.x86_64] Error occured, aborting > 10:33:23 [basic-suite.el7.x86_64] Traceback (most recent call last): > 10:33:23 [basic-suite.el7.x86_64] File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 383, in do_run > 10:33:23 [basic-suite.el7.x86_64] self.cli_plugins[args.ovirtverb].do_run(args) > 10:33:23 [basic-suite.el7.x86_64] File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run > 10:33:23 [basic-suite.el7.x86_64] self._do_run(**vars(args)) > 10:33:23 [basic-suite.el7.x86_64] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 505, in wrapper > 10:33:23 [basic-suite.el7.x86_64] return func(*args, **kwargs) > 10:33:23 [basic-suite.el7.x86_64] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 516, in wrapper > 10:33:23 [basic-suite.el7.x86_64] return func(*args, prefix=prefix, **kwargs) > 10:33:23 [basic-suite.el7.x86_64] File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 173, in do_ovirt_reposetup > 10:33:23 [basic-suite.el7.x86_64] sub_repo_name=repo_name > 10:33:23 [basic-suite.el7.x86_64] File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636, in wrapper > 10:33:23 [basic-suite.el7.x86_64] return func(*args, **kwargs) > 10:33:23 [basic-suite.el7.x86_64] File "/usr/lib/python2.7/site-packages/ovirtlago/prefix.py", line 203, in prepare_repo > 10:33:23 [basic-suite.el7.x86_64] repos, > 10:33:23 [basic-suite.el7.x86_64] File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 206, in sync_rpm_repository > 10:33:23 [basic-suite.el7.x86_64] with LockFile(lock_path, timeout=180): > 10:33:23 [basic-suite.el7.x86_64] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 398, in __enter__ > 10:33:23 [basic-suite.el7.x86_64] self.timeout, > 10:33:23 [basic-suite.el7.x86_64] TimerException: ('Unable to acquire lock for %s in %s secs', u'/var/lib/lago/repolock', 180) > {noformat} -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JIRA] (OVIRT-2572) ovirt-provider-ovn fails in 4.2 due to lago's repolock
by Dafna Ron (oVirt JIRA) 09 Nov '18

09 Nov '18

Dafna Ron created OVIRT-2572: -------------------------------- Summary: ovirt-provider-ovn fails in 4.2 due to lago's repolock Key: OVIRT-2572 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2572 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Dafna Ron Assignee: infra build that failed: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3468/ basic suite was running on ovirt-srv21.phx.ovirt.org {noformat} 10:33:23 [basic-suite.el7.x86_64] @ Create prefix internal repo: ERROR (in 0:03:00) 10:33:23 [basic-suite.el7.x86_64] Error occured, aborting 10:33:23 [basic-suite.el7.x86_64] Traceback (most recent call last): 10:33:23 [basic-suite.el7.x86_64] File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 383, in do_run 10:33:23 [basic-suite.el7.x86_64] self.cli_plugins[args.ovirtverb].do_run(args) 10:33:23 [basic-suite.el7.x86_64] File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run 10:33:23 [basic-suite.el7.x86_64] self._do_run(**vars(args)) 10:33:23 [basic-suite.el7.x86_64] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 505, in wrapper 10:33:23 [basic-suite.el7.x86_64] return func(*args, **kwargs) 10:33:23 [basic-suite.el7.x86_64] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 516, in wrapper 10:33:23 [basic-suite.el7.x86_64] return func(*args, prefix=prefix, **kwargs) 10:33:23 [basic-suite.el7.x86_64] File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 173, in do_ovirt_reposetup 10:33:23 [basic-suite.el7.x86_64] sub_repo_name=repo_name 10:33:23 [basic-suite.el7.x86_64] File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636, in wrapper 10:33:23 [basic-suite.el7.x86_64] return func(*args, **kwargs) 10:33:23 [basic-suite.el7.x86_64] File "/usr/lib/python2.7/site-packages/ovirtlago/prefix.py", line 203, in prepare_repo 10:33:23 [basic-suite.el7.x86_64] repos, 10:33:23 [basic-suite.el7.x86_64] File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 206, in sync_rpm_repository 10:33:23 [basic-suite.el7.x86_64] with LockFile(lock_path, timeout=180): 10:33:23 [basic-suite.el7.x86_64] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 398, in __enter__ 10:33:23 [basic-suite.el7.x86_64] self.timeout, 10:33:23 [basic-suite.el7.x86_64] TimerException: ('Unable to acquire lock for %s in %s secs', u'/var/lib/lago/repolock', 180) {noformat} -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[CQ]: 94930, 7 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 09 Nov '18

09 Nov '18

A system test invoked by the "ovirt-master" change queue including change 94930,7 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 95190,2 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95190,2 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/94930/7 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95190/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11097/

1 0

[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-master - Build # 543 - Failure!
by jenkins＠jenkins.phx.ovirt.org 09 Nov '18

09 Nov '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/543/ Build Number: 543 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #543 [Eitan Raviv] networking: sync networks across cluster ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 012_local_maintenance_sdk.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/012_local_maintenance_sdk.py", line 42, in local_maintenance time.sleep(wait_value) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'

1 1

[CQ]: 95280, 2 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 09 Nov '18

09 Nov '18

A system test invoked by the "ovirt-master" change queue including change 95280,2 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 95190,2 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95190,2 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/95280/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95190/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11093/

1 0

[CQ]: 95286, 3 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 09 Nov '18

09 Nov '18

A system test invoked by the "ovirt-master" change queue including change 95286,3 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 95190,2 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95190,2 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/95286/3 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95190/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11088/

1 0

[CQ]: 95322, 2 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 09 Nov '18

09 Nov '18

A system test invoked by the "ovirt-master" change queue including change 95322,2 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 95190,2 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95190,2 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/95322/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95190/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11086/

1 0

[CQ]: 95323, 2 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 09 Nov '18

09 Nov '18

A system test invoked by the "ovirt-master" change queue including change 95323,2 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 95190,2 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95190,2 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/95323/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95190/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11081/

1 0

[oVirt Jenkins] ovirt-system-tests_network-suite-master - Build # 741 - Failure!
by jenkins＠jenkins.phx.ovirt.org 09 Nov '18

09 Nov '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-master/741/ Build Number: 741 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #741 [Eitan Raviv] networking: sync networks across cluster ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: network-suite-master.tests.ovs.test_ovn_physnet.test_connect_vm_to_external_physnet Error Message: test setup failure Stack Trace: system = <lib.system.SDKSystemRoot object at 0x7f4ae66508d0> ovs_cluster = <lib.clusterlib.Cluster object at 0x7f4ad70fff10> default_cluster = <lib.clusterlib.Cluster object at 0x7f4ae6650b50> default_data_center = <lib.datacenterlib.DataCenter object at 0x7f4ad7bb0cd0> @pytest.fixture(scope='session') def host_in_ovs_cluster( system, ovs_cluster, default_cluster, default_data_center): host_id = default_cluster.host_ids()[0] host = hostlib.Host(system) host.import_by_id(host_id) host.wait_for_up_status(timeout=hostlib.HOST_TIMEOUT_LONG) with hostlib.change_cluster(host, ovs_cluster): host.sync_all_networks() default_data_center.wait_for_up_status() > yield host network-suite-master/fixtures/host.py:73: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib64/python2.7/contextlib.py:35: in __exit__ self.gen.throw(type, value, traceback) network-suite-master/lib/hostlib.py:84: in change_cluster host.change_cluster(original_cluster) network-suite-master/lib/hostlib.py:141: in change_cluster self.deactivate() network-suite-master/lib/hostlib.py:138: in deactivate self.wait_for_maintenance_status() network-suite-master/lib/hostlib.py:230: in wait_for_maintenance_status success_criteria=lambda s: s == HostStatus.MAINTENANCE) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ exec_func = <function <lambda> at 0x7f4ad71b3050>, exec_func_args = () success_criteria = <function <lambda> at 0x7f4ad71b3848> error_criteria = <function <lambda> at 0x7f4aead18b90>, timeout = 120 def sync(exec_func, exec_func_args, success_criteria=lambda result: True, error_criteria=lambda error: True, timeout=DEFAULT_TIMEOUT): """Sync an operation until it either: - succeeds (according to the success_criteria specified) - fails due to timing out (after the specified timeout) - fails due to a terminal error (according to the error_criteria specified) A caller may specifiy a success_criteria function that should return: - False if the sync should continue to retry - True if the sync should terminate immediately A caller may also specify an error_criteria function if the sync should continue to retry when the operation fails with an anticipated exception. This function will be called back with the exception and should return: - False if the sync should continue to retry - True if the sync should stop and the exception raised back to the caller By default, both success_criteria and error_criteria return True, causing all results and all errors to return and raise respectively. The default timeout is 120 seconds. :param exec_func: callable :param exec_func_args: tuple/dict :param success_criteria: callable :param error_criteria: callable :param timeout: int :return: the result of running the exec_func """ end_time = _monothonic_time() + timeout if isinstance(exec_func_args, collections.Mapping): kwargs = exec_func_args args = () else: args = exec_func_args kwargs = {} try: result = exec_func(*args, **kwargs) except Exception as e: if error_criteria(e): raise result = e else: if success_criteria(result): return result while _monothonic_time() < end_time: time.sleep(3) try: result = exec_func(*args, **kwargs) except Exception as e: if error_criteria(e): raise result = e else: if success_criteria(result): return result > raise Timeout(result) E Timeout: Last evaluated result: up network-suite-master/lib/syncutil.py:104: Timeout

1 1

oVirt infra daily report - unstable production jobs - 638
by jenkins＠jenkins.phx.ovirt.org 09 Nov '18

09 Nov '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/638//artifact/exported-a… Cheers, Jenkins

1 0

[CQ]: 95282, 3 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 08 Nov '18

08 Nov '18

A system test invoked by the "ovirt-master" change queue including change 95282,3 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 95190,2 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95190,2 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/95282/3 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95190/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11078/

1 0

[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-4.2 - Build # 475 - Failure!
by jenkins＠jenkins.phx.ovirt.org 08 Nov '18

08 Nov '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/475/ Build Number: 475 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #475 [Roy Golan] Add README to openshift-on-ovirt suite ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 012_local_maintenance_sdk.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... root: INFO: * Performing Deactivation... root: INFO: * Performing Activation... root: INFO: * Waiting For System Stability... cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/012_local_maintenance_sdk.py", line 79, in local_maintenance time.sleep(wait_value) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nroot: INFO: * Performing Deactivation...\nroot: INFO: * Performing Activation...\nroot: INFO: * Waiting For System Stability...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'

1 1

[oVirt Jenkins] ovirt-system-tests_he-basic-ansible-suite-4.2 - Build # 730 - Failure!
by jenkins＠jenkins.phx.ovirt.org 08 Nov '18

08 Nov '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4.2/… Build Number: 730 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #730 [Roy Golan] Add README to openshift-on-ovirt suite ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 008_restart_he_vm.restart_he_vm Error Message: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2675 (Thu Nov 8 03:19:13 2018)\nhost-id=1\nscore=3111\nvm_conf_refresh_time=2675 (Thu Nov 8 03:19:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "be3d40a1", "local_conf_timestamp": 2675, "host-ts": 2675}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2673 (Thu Nov 8 03:19:12 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2673 (Thu Nov 8 03:19:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2 -------------------- >> begin captured logging << -------------------- lago.ssh: DEBUG: start task:053eb136-48b1-45d4-a8b7-1b6f74536f40:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:053eb136-48b1-45d4-a8b7-1b6f74536f40:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running b6cf4628 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b6cf4628 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command b6cf4628 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2555 (Thu Nov 8 03:17:13 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2555 (Thu Nov 8 03:17:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "91a2a003", "local_conf_timestamp": 2555, "host-ts": 2555}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2553 (Thu Nov 8 03:17:12 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2553 (Thu Nov 8 03:17:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a7e651e6", "local_conf_timestamp": 2553, "host-ts": 2553}, "global_maintenance": true} root: INFO: * Shutting down HE VM on host: lago-he-basic-ansible-suite-4-2-host-0 lago.ssh: DEBUG: start task:48e52f8d-78cd-4c3e-9bdb-f862ac50a1e5:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:48e52f8d-78cd-4c3e-9bdb-f862ac50a1e5:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running b7683c5c on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-shutdown lago.ssh: DEBUG: Command b7683c5c on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 root: INFO: * Command succeeded root: INFO: * Waiting for VM to be down... lago.ssh: DEBUG: start task:b8b1cea3-0cce-43d1-8872-84dc6282c96d:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:b8b1cea3-0cce-43d1-8872-84dc6282c96d:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running b83d8ef2 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b83d8ef2 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command b83d8ef2 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2555 (Thu Nov 8 03:17:13 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2555 (Thu Nov 8 03:17:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "91a2a003", "local_conf_timestamp": 2555, "host-ts": 2555}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2553 (Thu Nov 8 03:17:12 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2553 (Thu Nov 8 03:17:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a7e651e6", "local_conf_timestamp": 2553, "host-ts": 2553}, "global_maintenance": true} lago.ssh: DEBUG: start task:e6d043c5-156b-4288-a7a3-f24161654e22:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:e6d043c5-156b-4288-a7a3-f24161654e22:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running bed30274 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command bed30274 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command bed30274 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2575 (Thu Nov 8 03:17:33 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2575 (Thu Nov 8 03:17:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "73a2b866", "local_conf_timestamp": 2575, "host-ts": 2575}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2573 (Thu Nov 8 03:17:31 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2573 (Thu Nov 8 03:17:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "904ac61d", "local_conf_timestamp": 2573, "host-ts": 2573}, "global_maintenance": true} root: INFO: * VM is down. root: INFO: * Stopping services... lago.ssh: DEBUG: start task:4d492e84-b98f-410a-9186-9da4ad96953b:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:4d492e84-b98f-410a-9186-9da4ad96953b:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running bf583408 on lago-he-basic-ansible-suite-4-2-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command bf583408 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 root: INFO: * Starting services... lago.ssh: DEBUG: start task:570131db-44f3-4f33-9b1c-1ca470f7db23:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:570131db-44f3-4f33-9b1c-1ca470f7db23:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running c7ea8f58 on lago-he-basic-ansible-suite-4-2-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command c7ea8f58 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 root: INFO: * Waiting for agent to be ready... lago.ssh: DEBUG: start task:15866098-59b0-4b3e-aca7-9f87b0faaa09:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:15866098-59b0-4b3e-aca7-9f87b0faaa09:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running ca1f0a7e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command ca1f0a7e on lago-he-basic-ansible-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command ca1f0a7e on lago-he-basic-ansible-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:9d69a85b-cf95-4cd5-acf7-849b3eef94ae:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:9d69a85b-cf95-4cd5-acf7-849b3eef94ae:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running d06eaa7e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command d06eaa7e on lago-he-basic-ansible-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command d06eaa7e on lago-he-basic-ansible-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:f122fda8-a80e-420d-8427-bd667eaf09c3:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:f122fda8-a80e-420d-8427-bd667eaf09c3:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running d2bcad8a on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command d2bcad8a on lago-he-basic-ansible-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command d2bcad8a on lago-he-basic-ansible-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:4f32a250-ab55-46c5-b228-a5ec3477a39c:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:4f32a250-ab55-46c5-b228-a5ec3477a39c:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running d4e22d24 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command d4e22d24 on lago-he-basic-ansible-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command d4e22d24 on lago-he-basic-ansible-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:fa589610-73ae-4944-9b08-cfcc0c7a3256:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:fa589610-73ae-4944-9b08-cfcc0c7a3256:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running d708a61e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command d708a61e on lago-he-basic-ansible-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command d708a61e on lago-he-basic-ansible-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:31625e1e-23bb-43b9-9ab7-910a7dafc5ed:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:31625e1e-23bb-43b9-9ab7-910a7dafc5ed:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running d93645ae on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command d93645ae on lago-he-basic-ansible-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command d93645ae on lago-he-basic-ansible-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:6077407c-ae81-4203-bfe4-007bae8817fb:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:6077407c-ae81-4203-bfe4-007bae8817fb:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running db5f2030 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command db5f2030 on lago-he-basic-ansible-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command db5f2030 on lago-he-basic-ansible-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:77846988-89e4-45f0-8aba-cacb985bf1b9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:77846988-89e4-45f0-8aba-cacb985bf1b9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running dd889382 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command dd889382 on lago-he-basic-ansible-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command dd889382 on lago-he-basic-ansible-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:f1925127-d380-43cc-86ea-4f90bcb5b63d:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:f1925127-d380-43cc-86ea-4f90bcb5b63d:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running dfb22bbe on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command dfb22bbe on lago-he-basic-ansible-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command dfb22bbe on lago-he-basic-ansible-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:4edc99b2-4a65-4147-b24d-56bbc8046b66:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:4edc99b2-4a65-4147-b24d-56bbc8046b66:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running e1d9e5e4 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command e1d9e5e4 on lago-he-basic-ansible-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command e1d9e5e4 on lago-he-basic-ansible-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:70bfaba1-ad03-4f96-918f-1e2793ee7427:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:70bfaba1-ad03-4f96-918f-1e2793ee7427:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running e40a02a4 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command e40a02a4 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command e40a02a4 on lago-he-basic-ansible-suite-4-2-host-0 output: !! Cluster is in GLOBAL MAINTENANCE mode !! --== Host 1 status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : lago-he-basic-ansible-suite-4-2-host-0 Host ID : 1 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"} Score : 0 stopped : False Local maintenance : False crc32 : 36bd3d62 local_conf_timestamp : 2635 Host timestamp : 2635 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2635 (Thu Nov 8 03:18:33 2018) host-id=1 score=0 vm_conf_refresh_time=2635 (Thu Nov 8 03:18:33 2018) conf_on_shared_storage=True maintenance=False state=ReinitializeFSM stopped=False --== Host 2 status ==-- conf_on_shared_storage : True Status up-to-date : False Hostname : lago-he-basic-ansible-suite-4-2-host-1 Host ID : 2 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : 2ddfae49 local_conf_timestamp : 2633 Host timestamp : 2633 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2633 (Thu Nov 8 03:18:32 2018) host-id=2 score=3400 vm_conf_refresh_time=2633 (Thu Nov 8 03:18:32 2018) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False !! Cluster is in GLOBAL MAINTENANCE mode !! root: INFO: * Agent is ready. root: INFO: * Starting VM... lago.ssh: DEBUG: start task:ff98dc0f-e7fc-4cb8-b035-7af590da59dd:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:ff98dc0f-e7fc-4cb8-b035-7af590da59dd:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running e4911c3a on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-start lago.ssh: DEBUG: Command e4911c3a on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command e4911c3a on lago-he-basic-ansible-suite-4-2-host-0 output: VM exists and is down, cleaning up and restarting root: INFO: * Command succeeded root: INFO: * Waiting for VM to be UP... lago.ssh: DEBUG: start task:acf835ef-f2f1-4150-9e4a-69f3b18687fa:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:acf835ef-f2f1-4150-9e4a-69f3b18687fa:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running e5c3aa46 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e5c3aa46 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command e5c3aa46 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2635 (Thu Nov 8 03:18:33 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=2635 (Thu Nov 8 03:18:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=ReinitializeFSM\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "36bd3d62", "local_conf_timestamp": 2635, "host-ts": 2635}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2633 (Thu Nov 8 03:18:32 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2633 (Thu Nov 8 03:18:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2ddfae49", "local_conf_timestamp": 2633, "host-ts": 2633}, "global_maintenance": true} lago.ssh: DEBUG: start task:eb7dba65-0edb-4c63-9c01-22e619db3e60:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:eb7dba65-0edb-4c63-9c01-22e619db3e60:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running ec372e3e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ec372e3e on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command ec372e3e on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2645 (Thu Nov 8 03:18:44 2018)\nhost-id=1\nscore=3111\nvm_conf_refresh_time=2645 (Thu Nov 8 03:18:44 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "0df3ca26", "local_conf_timestamp": 2645, "host-ts": 2645}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2643 (Thu Nov 8 03:18:42 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2643 (Thu Nov 8 03:18:42 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8344de99", "local_conf_timestamp": 2643, "host-ts": 2643}, "global_maintenance": true} root: INFO: * VM is UP. root: INFO: * Waiting for engine to start... lago.ssh: DEBUG: start task:8bf6658f-e1f4-4aa8-a773-b2d92f60e02a:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:8bf6658f-e1f4-4aa8-a773-b2d92f60e02a:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running ecaca8ee on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ecaca8ee on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command ecaca8ee on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2645 (Thu Nov 8 03:18:44 2018)\nhost-id=1\nscore=3111\nvm_conf_refresh_time=2645 (Thu Nov 8 03:18:44 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "0df3ca26", "local_conf_timestamp": 2645, "host-ts": 2645}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2643 (Thu Nov 8 03:18:42 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2643 (Thu Nov 8 03:18:42 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8344de99", "local_conf_timestamp": 2643, "host-ts": 2643}, "global_maintenance": true} lago.ssh: DEBUG: start task:71c5108d-271c-4aad-8aec-0a8b7337f359:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:71c5108d-271c-4aad-8aec-0a8b7337f359:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running f330d5aa on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f330d5aa on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command f330d5aa on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2655 (Thu Nov 8 03:18:54 2018)\nhost-id=1\nscore=3111\nvm_conf_refresh_time=2656 (Thu Nov 8 03:18:54 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "4c901fde", "local_conf_timestamp": 2656, "host-ts": 2655}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2653 (Thu Nov 8 03:18:51 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2653 (Thu Nov 8 03:18:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "752a1644", "local_conf_timestamp": 2653, "host-ts": 2653}, "global_maintenance": true} lago.ssh: DEBUG: start task:e68357ea-ea52-44f4-b67e-e22f47d6a2ac:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:e68357ea-ea52-44f4-b67e-e22f47d6a2ac:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running f5a2235c on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f5a2235c on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command f5a2235c on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2665 (Thu Nov 8 03:19:03 2018)\nhost-id=1\nscore=3111\nvm_conf_refresh_time=2665 (Thu Nov 8 03:19:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "6657c4e3", "local_conf_timestamp": 2665, "host-ts": 2665}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2663 (Thu Nov 8 03:19:02 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2663 (Thu Nov 8 03:19:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b4e84962", "local_conf_timestamp": 2663, "host-ts": 2663}, "global_maintenance": true} lago.ssh: DEBUG: start task:436dfa11-97d6-4095-804e-566329a5fdee:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:436dfa11-97d6-4095-804e-566329a5fdee:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running f7ff723a on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f7ff723a on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command f7ff723a on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2665 (Thu Nov 8 03:19:03 2018)\nhost-id=1\nscore=3111\nvm_conf_refresh_time=2665 (Thu Nov 8 03:19:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "6657c4e3", "local_conf_timestamp": 2665, "host-ts": 2665}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2663 (Thu Nov 8 03:19:02 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2663 (Thu Nov 8 03:19:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b4e84962", "local_conf_timestamp": 2663, "host-ts": 2663}, "global_maintenance": true} lago.ssh: DEBUG: start task:de0e96fe-575e-4a46-82f8-4e3d5a85ca1f:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:de0e96fe-575e-4a46-82f8-4e3d5a85ca1f:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running fa442a36 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command fa442a36 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command fa442a36 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2675 (Thu Nov 8 03:19:13 2018)\nhost-id=1\nscore=3111\nvm_conf_refresh_time=2675 (Thu Nov 8 03:19:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "be3d40a1", "local_conf_timestamp": 2675, "host-ts": 2675}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2673 (Thu Nov 8 03:19:12 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2673 (Thu Nov 8 03:19:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "428681bf", "local_conf_timestamp": 2673, "host-ts": 2673}, "global_maintenance": true} lago.ssh: DEBUG: start task:97f350dc-f503-4726-84cb-61ad3b8166bc:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:97f350dc-f503-4726-84cb-61ad3b8166bc:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running fc9a48b0 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command fc9a48b0 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command fc9a48b0 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2675 (Thu Nov 8 03:19:13 2018)\nhost-id=1\nscore=3111\nvm_conf_refresh_time=2675 (Thu Nov 8 03:19:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "be3d40a1", "local_conf_timestamp": 2675, "host-ts": 2675}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2673 (Thu Nov 8 03:19:12 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2673 (Thu Nov 8 03:19:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2 ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f1389d43e60> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/008_restart_he_vm.py", line 187, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) RuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2675 (Thu Nov 8 03:19:13 2018)\nhost-id=1\nscore=3111\nvm_conf_refresh_time=2675 (Thu Nov 8 03:19:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "be3d40a1", "local_conf_timestamp": 2675, "host-ts": 2675}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2673 (Thu Nov 8 03:19:12 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2673 (Thu Nov 8 03:19:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2 --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/008_restart_he_vm.py", line 53, in restart_he_vm _wait_for_engine_health(host) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/008_restart_he_vm.py", line 185, in _wait_for_engine_health testlib.assert_true_within_long(lambda: any( File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/008_restart_he_vm.py", line 187, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) 'could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2675 (Thu Nov 8 03:19:13 2018)\\nhost-id=1\\nscore=3111\\nvm_conf_refresh_time=2675 (Thu Nov 8 03:19:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "be3d40a1", "local_conf_timestamp": 2675, "host-ts": 2675}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2673 (Thu Nov 8 03:19:12 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2673 (Thu Nov 8 03:19:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:053eb136-48b1-45d4-a8b7-1b6f74536f40:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:053eb136-48b1-45d4-a8b7-1b6f74536f40:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b6cf4628 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b6cf4628 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command b6cf4628 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2555 (Thu Nov 8 03:17:13 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2555 (Thu Nov 8 03:17:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "91a2a003", "local_conf_timestamp": 2555, "host-ts": 2555}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2553 (Thu Nov 8 03:17:12 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2553 (Thu Nov 8 03:17:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a7e651e6", "local_conf_timestamp": 2553, "host-ts": 2553}, "global_maintenance": true}\n\nroot: INFO: * Shutting down HE VM on host: lago-he-basic-ansible-suite-4-2-host-0\nlago.ssh: DEBUG: start task:48e52f8d-78cd-4c3e-9bdb-f862ac50a1e5:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:48e52f8d-78cd-4c3e-9bdb-f862ac50a1e5:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b7683c5c on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-shutdown\nlago.ssh: DEBUG: Command b7683c5c on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be down...\nlago.ssh: DEBUG: start task:b8b1cea3-0cce-43d1-8872-84dc6282c96d:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:b8b1cea3-0cce-43d1-8872-84dc6282c96d:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b83d8ef2 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b83d8ef2 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command b83d8ef2 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2555 (Thu Nov 8 03:17:13 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2555 (Thu Nov 8 03:17:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "91a2a003", "local_conf_timestamp": 2555, "host-ts": 2555}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2553 (Thu Nov 8 03:17:12 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2553 (Thu Nov 8 03:17:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a7e651e6", "local_conf_timestamp": 2553, "host-ts": 2553}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:e6d043c5-156b-4288-a7a3-f24161654e22:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:e6d043c5-156b-4288-a7a3-f24161654e22:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running bed30274 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command bed30274 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command bed30274 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2575 (Thu Nov 8 03:17:33 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2575 (Thu Nov 8 03:17:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "73a2b866", "local_conf_timestamp": 2575, "host-ts": 2575}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2573 (Thu Nov 8 03:17:31 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2573 (Thu Nov 8 03:17:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "904ac61d", "local_conf_timestamp": 2573, "host-ts": 2573}, "global_maintenance": true}\n\nroot: INFO: * VM is down.\nroot: INFO: * Stopping services...\nlago.ssh: DEBUG: start task:4d492e84-b98f-410a-9186-9da4ad96953b:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:4d492e84-b98f-410a-9186-9da4ad96953b:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running bf583408 on lago-he-basic-ansible-suite-4-2-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command bf583408 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nroot: INFO: * Starting services...\nlago.ssh: DEBUG: start task:570131db-44f3-4f33-9b1c-1ca470f7db23:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:570131db-44f3-4f33-9b1c-1ca470f7db23:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running c7ea8f58 on lago-he-basic-ansible-suite-4-2-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command c7ea8f58 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nroot: INFO: * Waiting for agent to be ready...\nlago.ssh: DEBUG: start task:15866098-59b0-4b3e-aca7-9f87b0faaa09:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:15866098-59b0-4b3e-aca7-9f87b0faaa09:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running ca1f0a7e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command ca1f0a7e on lago-he-basic-ansible-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command ca1f0a7e on lago-he-basic-ansible-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:9d69a85b-cf95-4cd5-acf7-849b3eef94ae:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:9d69a85b-cf95-4cd5-acf7-849b3eef94ae:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running d06eaa7e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command d06eaa7e on lago-he-basic-ansible-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command d06eaa7e on lago-he-basic-ansible-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:f122fda8-a80e-420d-8427-bd667eaf09c3:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:f122fda8-a80e-420d-8427-bd667eaf09c3:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running d2bcad8a on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command d2bcad8a on lago-he-basic-ansible-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command d2bcad8a on lago-he-basic-ansible-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:4f32a250-ab55-46c5-b228-a5ec3477a39c:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:4f32a250-ab55-46c5-b228-a5ec3477a39c:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running d4e22d24 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command d4e22d24 on lago-he-basic-ansible-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command d4e22d24 on lago-he-basic-ansible-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:fa589610-73ae-4944-9b08-cfcc0c7a3256:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:fa589610-73ae-4944-9b08-cfcc0c7a3256:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running d708a61e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command d708a61e on lago-he-basic-ansible-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command d708a61e on lago-he-basic-ansible-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:31625e1e-23bb-43b9-9ab7-910a7dafc5ed:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:31625e1e-23bb-43b9-9ab7-910a7dafc5ed:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running d93645ae on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command d93645ae on lago-he-basic-ansible-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command d93645ae on lago-he-basic-ansible-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:6077407c-ae81-4203-bfe4-007bae8817fb:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:6077407c-ae81-4203-bfe4-007bae8817fb:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running db5f2030 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command db5f2030 on lago-he-basic-ansible-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command db5f2030 on lago-he-basic-ansible-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:77846988-89e4-45f0-8aba-cacb985bf1b9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:77846988-89e4-45f0-8aba-cacb985bf1b9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running dd889382 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command dd889382 on lago-he-basic-ansible-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command dd889382 on lago-he-basic-ansible-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:f1925127-d380-43cc-86ea-4f90bcb5b63d:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:f1925127-d380-43cc-86ea-4f90bcb5b63d:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running dfb22bbe on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command dfb22bbe on lago-he-basic-ansible-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command dfb22bbe on lago-he-basic-ansible-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:4edc99b2-4a65-4147-b24d-56bbc8046b66:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:4edc99b2-4a65-4147-b24d-56bbc8046b66:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running e1d9e5e4 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command e1d9e5e4 on lago-he-basic-ansible-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command e1d9e5e4 on lago-he-basic-ansible-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:70bfaba1-ad03-4f96-918f-1e2793ee7427:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:70bfaba1-ad03-4f96-918f-1e2793ee7427:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running e40a02a4 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command e40a02a4 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command e40a02a4 on lago-he-basic-ansible-suite-4-2-host-0 output:\n \n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\n\n--== Host 1 status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : True\nHostname : lago-he-basic-ansible-suite-4-2-host-0\nHost ID : 1\nEngine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}\nScore : 0\nstopped : False\nLocal maintenance : False\ncrc32 : 36bd3d62\nlocal_conf_timestamp : 2635\nHost timestamp : 2635\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2635 (Thu Nov 8 03:18:33 2018)\n\thost-id=1\n\tscore=0\n\tvm_conf_refresh_time=2635 (Thu Nov 8 03:18:33 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=ReinitializeFSM\n\tstopped=False\n\n\n--== Host 2 status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : False\nHostname : lago-he-basic-ansible-suite-4-2-host-1\nHost ID : 2\nEngine status : unknown stale-data\nScore : 3400\nstopped : False\nLocal maintenance : False\ncrc32 : 2ddfae49\nlocal_conf_timestamp : 2633\nHost timestamp : 2633\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2633 (Thu Nov 8 03:18:32 2018)\n\thost-id=2\n\tscore=3400\n\tvm_conf_refresh_time=2633 (Thu Nov 8 03:18:32 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=GlobalMaintenance\n\tstopped=False\n\n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\nroot: INFO: * Agent is ready.\nroot: INFO: * Starting VM...\nlago.ssh: DEBUG: start task:ff98dc0f-e7fc-4cb8-b035-7af590da59dd:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:ff98dc0f-e7fc-4cb8-b035-7af590da59dd:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running e4911c3a on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-start\nlago.ssh: DEBUG: Command e4911c3a on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command e4911c3a on lago-he-basic-ansible-suite-4-2-host-0 output:\n VM exists and is down, cleaning up and restarting\n\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be UP...\nlago.ssh: DEBUG: start task:acf835ef-f2f1-4150-9e4a-69f3b18687fa:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:acf835ef-f2f1-4150-9e4a-69f3b18687fa:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running e5c3aa46 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e5c3aa46 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command e5c3aa46 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2635 (Thu Nov 8 03:18:33 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=2635 (Thu Nov 8 03:18:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=ReinitializeFSM\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "36bd3d62", "local_conf_timestamp": 2635, "host-ts": 2635}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2633 (Thu Nov 8 03:18:32 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2633 (Thu Nov 8 03:18:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2ddfae49", "local_conf_timestamp": 2633, "host-ts": 2633}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:eb7dba65-0edb-4c63-9c01-22e619db3e60:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:eb7dba65-0edb-4c63-9c01-22e619db3e60:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running ec372e3e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ec372e3e on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command ec372e3e on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2645 (Thu Nov 8 03:18:44 2018)\\nhost-id=1\\nscore=3111\\nvm_conf_refresh_time=2645 (Thu Nov 8 03:18:44 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "0df3ca26", "local_conf_timestamp": 2645, "host-ts": 2645}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2643 (Thu Nov 8 03:18:42 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2643 (Thu Nov 8 03:18:42 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8344de99", "local_conf_timestamp": 2643, "host-ts": 2643}, "global_maintenance": true}\n\nroot: INFO: * VM is UP.\nroot: INFO: * Waiting for engine to start...\nlago.ssh: DEBUG: start task:8bf6658f-e1f4-4aa8-a773-b2d92f60e02a:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:8bf6658f-e1f4-4aa8-a773-b2d92f60e02a:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running ecaca8ee on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ecaca8ee on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command ecaca8ee on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2645 (Thu Nov 8 03:18:44 2018)\\nhost-id=1\\nscore=3111\\nvm_conf_refresh_time=2645 (Thu Nov 8 03:18:44 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "0df3ca26", "local_conf_timestamp": 2645, "host-ts": 2645}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2643 (Thu Nov 8 03:18:42 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2643 (Thu Nov 8 03:18:42 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8344de99", "local_conf_timestamp": 2643, "host-ts": 2643}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:71c5108d-271c-4aad-8aec-0a8b7337f359:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:71c5108d-271c-4aad-8aec-0a8b7337f359:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running f330d5aa on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f330d5aa on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command f330d5aa on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2655 (Thu Nov 8 03:18:54 2018)\\nhost-id=1\\nscore=3111\\nvm_conf_refresh_time=2656 (Thu Nov 8 03:18:54 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "4c901fde", "local_conf_timestamp": 2656, "host-ts": 2655}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2653 (Thu Nov 8 03:18:51 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2653 (Thu Nov 8 03:18:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "752a1644", "local_conf_timestamp": 2653, "host-ts": 2653}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:e68357ea-ea52-44f4-b67e-e22f47d6a2ac:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:e68357ea-ea52-44f4-b67e-e22f47d6a2ac:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running f5a2235c on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f5a2235c on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command f5a2235c on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2665 (Thu Nov 8 03:19:03 2018)\\nhost-id=1\\nscore=3111\\nvm_conf_refresh_time=2665 (Thu Nov 8 03:19:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "6657c4e3", "local_conf_timestamp": 2665, "host-ts": 2665}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2663 (Thu Nov 8 03:19:02 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2663 (Thu Nov 8 03:19:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b4e84962", "local_conf_timestamp": 2663, "host-ts": 2663}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:436dfa11-97d6-4095-804e-566329a5fdee:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:436dfa11-97d6-4095-804e-566329a5fdee:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running f7ff723a on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f7ff723a on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command f7ff723a on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2665 (Thu Nov 8 03:19:03 2018)\\nhost-id=1\\nscore=3111\\nvm_conf_refresh_time=2665 (Thu Nov 8 03:19:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "6657c4e3", "local_conf_timestamp": 2665, "host-ts": 2665}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2663 (Thu Nov 8 03:19:02 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2663 (Thu Nov 8 03:19:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b4e84962", "local_conf_timestamp": 2663, "host-ts": 2663}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:de0e96fe-575e-4a46-82f8-4e3d5a85ca1f:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:de0e96fe-575e-4a46-82f8-4e3d5a85ca1f:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running fa442a36 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fa442a36 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command fa442a36 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2675 (Thu Nov 8 03:19:13 2018)\\nhost-id=1\\nscore=3111\\nvm_conf_refresh_time=2675 (Thu Nov 8 03:19:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "be3d40a1", "local_conf_timestamp": 2675, "host-ts": 2675}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2673 (Thu Nov 8 03:19:12 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2673 (Thu Nov 8 03:19:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "428681bf", "local_conf_timestamp": 2673, "host-ts": 2673}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:97f350dc-f503-4726-84cb-61ad3b8166bc:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:97f350dc-f503-4726-84cb-61ad3b8166bc:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running fc9a48b0 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fc9a48b0 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command fc9a48b0 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2675 (Thu Nov 8 03:19:13 2018)\\nhost-id=1\\nscore=3111\\nvm_conf_refresh_time=2675 (Thu Nov 8 03:19:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "be3d40a1", "local_conf_timestamp": 2675, "host-ts": 2675}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2673 (Thu Nov 8 03:19:12 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2673 (Thu Nov 8 03:19:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f1389d43e60>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/008_restart_he_vm.py", line 187, in <lambda>\n for k, v in _get_he_status(host).items()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status\n raise RuntimeError(\'could not parse JSON: %s\' % ret.out)\nRuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2675 (Thu Nov 8 03:19:13 2018)\\nhost-id=1\\nscore=3111\\nvm_conf_refresh_time=2675 (Thu Nov 8 03:19:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "be3d40a1", "local_conf_timestamp": 2675, "host-ts": 2675}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2673 (Thu Nov 8 03:19:12 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2673 (Thu Nov 8 03:19:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2\n--------------------- >> end captured logging << ---------------------'

1 1

[CQ]: 95290, 4 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 08 Nov '18

08 Nov '18

A system test invoked by the "ovirt-master" change queue including change 95290,4 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 95190,2 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95190,2 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/95290/4 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95190/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11072/

1 0

[CQ]: 95163, 8 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 08 Nov '18

08 Nov '18

A system test invoked by the "ovirt-master" change queue including change 95163,8 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 95190,2 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95190,2 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/95163/8 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95190/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11066/

2 1

[JIRA] (OVIRT-2571) master ovirt-engine fails on basic suite, test remove_vm_pool
by Dafna Ron (oVirt JIRA) 08 Nov '18

08 Nov '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2571?page=com.atlassian.jira.… ] Dafna Ron reassigned OVIRT-2571: -------------------------------- Assignee: Dominik Holler (was: infra) > master ovirt-engine fails on basic suite, test remove_vm_pool > ------------------------------------------------------------- > > Key: OVIRT-2571 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2571 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: Dominik Holler > Labels: ost_failures > > This actually seems to me to be related to the test and not the patch. > We try to remove pool while vm is being updated. > seems like this test might be a network test they should look into this issue. > Will ask Dominic to look at this first and rule out his patch as the cause. > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11066/ > Patch that failed: https://gerrit.ovirt.org/#/c/95163/ > {noformat} > 2018-11-08 04:26:42,081-05 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-1) [] Operation Failed: [Cannot remove VM-Pool. VM is being updated.] > {noformat} -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JIRA] (OVIRT-2571) master ovirt-engine fails on basic suite, test remove_vm_pool
by Dafna Ron (oVirt JIRA) 08 Nov '18

08 Nov '18

Dafna Ron created OVIRT-2571: -------------------------------- Summary: master ovirt-engine fails on basic suite, test remove_vm_pool Key: OVIRT-2571 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2571 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Dafna Ron Assignee: infra This actually seems to me to be related to the test and not the patch. We try to remove pool while vm is being updated. seems like this test might be a network test they should look into this issue. Will ask Dominic to look at this first and rule out his patch as the cause. http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11066/ Patch that failed: https://gerrit.ovirt.org/#/c/95163/ {noformat} 2018-11-08 04:26:42,081-05 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-1) [] Operation Failed: [Cannot remove VM-Pool. VM is being updated.] {noformat} -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-master - Build # 540 - Failure!
by jenkins＠jenkins.phx.ovirt.org 08 Nov '18

08 Nov '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/540/ Build Number: 540 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #540 [Roy Golan] Add README to openshift-on-ovirt suite ----------------- Failed Tests: ----------------- No tests ran.

1 1

[oVirt CI] standard-enqueue - Build #17315 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 08 Nov '18

08 Nov '18

Build: http://jenkins.ovirt.org/job/standard-enqueue/17315/ Build Name: #17315 Build Description: Gerrit: 95320 - ovirt-engine (4.2.7) Build Status: FAILURE Gerrit change: https://gerrit.ovirt.org/95320 - title: engine: Adding missing volume options - project: ovirt-engine - branch: ovirt-engine-4.2.7.z - author: Sahina Bose <sabose(a)redhat.com>

1 0

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-4.2 - Build # 637 - Failure!
by jenkins＠jenkins.phx.ovirt.org 08 Nov '18

08 Nov '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/637/ Build Number: 637 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #637 [Roy Golan] Add README to openshift-on-ovirt suite ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.add_hosts Error Message: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML -------------------- >> begin captured logging << -------------------- ovirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7f1af0c9d578> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 254, in _host_is_up_4 host_obj = host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37093, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 229, in add_hosts add_hosts_4(prefix) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 273, in add_hosts_4 testlib.assert_true_within(_host_is_up_4, timeout=15*60) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 278, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 254, in _host_is_up_4 host_obj = host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37093, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type raise Error(msg) 'The response content type \'text/html; charset=iso-8859-1\' isn\'t the expected XML\n-------------------- >> begin captured logging << --------------------\novirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7f1af0c9d578>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 254, in _host_is_up_4\n host_obj = host_service.get()\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37093, in get\n return self._internal_get(headers, query, wait)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get\n return future.wait() if wait else future\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait\n return self._code(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback\n self._check_fault(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault\n body = self._internal_read_body(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body\n self._connection.check_xml_content_type(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type\n response.headers\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type\n raise Error(msg)\nError: The response content type \'text/html; charset=iso-8859-1\' isn\'t the expected XML\n--------------------- >> end captured logging << ---------------------'

1 1

[oVirt Jenkins] ovirt-node-ng-image_master_build-artifacts-fc28-x86_64 - Build # 101 - Failure!
by jenkins＠jenkins.phx.ovirt.org 08 Nov '18

08 Nov '18

Project: http://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-fc2… Build: http://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-fc2… Build Number: 101 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #101 [Yuval Turgeman] stdci: move build-artifacts to a v1 job ----------------- Failed Tests: ----------------- No tests ran.

1 9

[CQ]: 95180,8 (vdsm) failed "ovirt-master" system tests
by oVirt Jenkins 08 Nov '18

08 Nov '18

Change 95180,8 (vdsm) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/95180/8 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11062/

2 1

[CQ]: 95130, 5 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 08 Nov '18

08 Nov '18

A system test invoked by the "ovirt-master" change queue including change 95130,5 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 95190,2 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95190,2 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/95130/5 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95190/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11058/

1 0

oVirt infra daily report - unstable production jobs - 637
by jenkins＠jenkins.phx.ovirt.org 08 Nov '18

08 Nov '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/637//artifact/exported-a… Cheers, Jenkins

1 0

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build # 856 - Failure!
by jenkins＠jenkins.phx.ovirt.org 07 Nov '18

07 Nov '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/856/ Build Number: 856 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #856 [Roy Golan] Fetch the quay.io image for deployment ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.add_hosts Error Message: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML -------------------- >> begin captured logging << -------------------- ovirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7f8c98e60320> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4 host_obj = host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 38639, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 850, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 890, in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 228, in add_hosts add_hosts_4(prefix) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 275, in add_hosts_4 testlib.assert_true_within(_host_is_up_4, timeout=15*60) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 278, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4 host_obj = host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 38639, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 850, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 890, in _check_content_type raise Error(msg) 'The response content type \'text/html; charset=iso-8859-1\' isn\'t the expected XML\n-------------------- >> begin captured logging << --------------------\novirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7f8c98e60320>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4\n host_obj = host_service.get()\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 38639, in get\n return self._internal_get(headers, query, wait)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get\n return future.wait() if wait else future\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait\n return self._code(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback\n self._check_fault(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault\n body = self._internal_read_body(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body\n self._connection.check_xml_content_type(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 850, in check_xml_content_type\n response.headers\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 890, in _check_content_type\n raise Error(msg)\nError: The response content type \'text/html; charset=iso-8859-1\' isn\'t the expected XML\n--------------------- >> end captured logging << ---------------------'

1 1

[CQ]: 95251, 4 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 07 Nov '18

07 Nov '18

A system test invoked by the "ovirt-master" change queue including change 95251,4 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 95190,2 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95190,2 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/95251/4 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95190/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11052/

1 0

[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-4.2 - Build # 471 - Failure!
by jenkins＠jenkins.phx.ovirt.org 07 Nov '18

07 Nov '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/471/ Build Number: 471 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #471 [Galit Rosenthal] Added PG10 to master repo file ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 012_local_maintenance_sdk.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... root: INFO: * Performing Deactivation... root: INFO: * Performing Activation... root: INFO: * Waiting For System Stability... cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/012_local_maintenance_sdk.py", line 79, in local_maintenance time.sleep(wait_value) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nroot: INFO: * Performing Deactivation...\nroot: INFO: * Performing Activation...\nroot: INFO: * Waiting For System Stability...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'

1 2

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-4.2 - Build # 635 - Failure!
by jenkins＠jenkins.phx.ovirt.org 07 Nov '18

07 Nov '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/635/ Build Number: 635 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #635 [Eitan Raviv] networking: introduce Node with ssh capabilities ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.add_hosts Error Message: Failed to read response: [(<pycurl.Curl object at 0x7f36c6c8b050>, 7, 'TCP connection reset by peer')] -------------------- >> begin captured logging << -------------------- ovirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7f36c6c2c578> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 254, in _host_is_up_4 host_obj = host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37093, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 54, in wait response = self._connection.wait(self._context) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 496, in wait return self.__wait(context, failed_auth) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 510, in __wait raise Error("Failed to read response: {}".format(err_list)) Error: Failed to read response: [(<pycurl.Curl object at 0x7f36c6c8b050>, 7, 'TCP connection reset by peer')] --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 229, in add_hosts add_hosts_4(prefix) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 273, in add_hosts_4 testlib.assert_true_within(_host_is_up_4, timeout=15*60) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 278, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 254, in _host_is_up_4 host_obj = host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37093, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 54, in wait response = self._connection.wait(self._context) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 496, in wait return self.__wait(context, failed_auth) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 510, in __wait raise Error("Failed to read response: {}".format(err_list)) 'Failed to read response: [(<pycurl.Curl object at 0x7f36c6c8b050>, 7, \'TCP connection reset by peer\')]\n-------------------- >> begin captured logging << --------------------\novirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7f36c6c2c578>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 254, in _host_is_up_4\n host_obj = host_service.get()\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37093, in get\n return self._internal_get(headers, query, wait)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get\n return future.wait() if wait else future\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 54, in wait\n response = self._connection.wait(self._context)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 496, in wait\n return self.__wait(context, failed_auth)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 510, in __wait\n raise Error("Failed to read response: {}".format(err_list))\nError: Failed to read response: [(<pycurl.Curl object at 0x7f36c6c8b050>, 7, \'TCP connection reset by peer\')]\n--------------------- >> end captured logging << ---------------------'

1 1

[CQ]: 95270, 2 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 07 Nov '18

07 Nov '18

A system test invoked by the "ovirt-master" change queue including change 95270,2 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 95190,2 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95190,2 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/95270/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95190/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11048/

1 0