- Infra - oVirt List Archives

[CQ]: 92516,21 (vdsm) failed "ovirt-master" system tests
by oVirt Jenkins 26 Jul '18

26 Jul '18

Change 92516,21 (vdsm) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/92516/21 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8810/

1 0

[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-master - Build # 226 - Failure!
by jenkins＠jenkins.phx.ovirt.org 26 Jul '18

26 Jul '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/226/ Build Number: 226 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #226 [Gal Ben Haim] compat: Set "ex" bash flags [Barak Korren] Remove VDSM build for s390x [Barak Korren] global_setup: Verify the DOCKER iptables chain [Barak Korren] STDCI: Archive artifacts earlier [Barak Korren] STDCI Report: Near real-time updating ----------------- Failed Tests: ----------------- No tests ran.

1 1

[oVirt Jenkins] ovirt-system-tests_he-basic-iscsi-suite-4.2 - Build # 160 - Failure!
by jenkins＠jenkins.phx.ovirt.org 26 Jul '18

26 Jul '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-4.2/16… Build Number: 160 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #160 [Gal Ben Haim] compat: Set "ex" bash flags [Barak Korren] Remove VDSM build for s390x [Barak Korren] global_setup: Verify the DOCKER iptables chain [Barak Korren] STDCI: Archive artifacts earlier [Barak Korren] STDCI Report: Near real-time updating ----------------- Failed Tests: ----------------- No tests ran.

1 1

[CQ]: 2dbaf1d (ovirt-ansible-vm-infra) failed "ovirt-master" system tests
by oVirt Jenkins 26 Jul '18

26 Jul '18

Change 2dbaf1d (ovirt-ansible-vm-infra) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://github.com/oVirt/ovirt-ansible-vm-infra/commit/2dbaf1d69499b65ecd7a… For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8800/

1 0

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build # 605 - Failure!
by jenkins＠jenkins.phx.ovirt.org 26 Jul '18

26 Jul '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/605/ Build Number: 605 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #605 [Gal Ben Haim] compat: Set "ex" bash flags [Yuval Turgeman] Add ovirt-node-ng-image project to stdci-v2 ----------------- Failed Tests: ----------------- No tests ran.

1 2

Build failed in Jenkins: system-sync_mirrors-epel-el6-x86_64 #1707
by jenkins＠jenkins.phx.ovirt.org 26 Jul '18

26 Jul '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/1707/displ…> Changes: [Barak Korren] Remove VDSM build for s390x [Barak Korren] global_setup: Verify the DOCKER iptables chain [Barak Korren] STDCI: Archive artifacts earlier [Barak Korren] STDCI Report: Near real-time updating ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision bcb1ae15c71f76699619e67e736e77531cd14a14 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f bcb1ae15c71f76699619e67e736e77531cd14a14 Commit message: "STDCI Report: Near real-time updating" > git rev-list --no-walk 5d4500574ebd94d19c89b38fd1d96065e4e99560 # timeout=10 [system-sync_mirrors-epel-el6-x86_64] $ /bin/bash -xe /tmp/jenkins4988293610991311087.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror epel-el6 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 209, in main my.doSackSetup(arches) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 741, in doSackSetup return self._getSacks(archlist=archlist, thisrepo=thisrepo) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 778, in _getSacks self.repos.populateSack(which=repos) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 386, in populateSack sack.populate(repo, mdtype, callback, cacheonly) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 256, in populate ret = misc.repo_gen_decompress(xml, gen, cached=repo.cache) File "/usr/lib/python2.7/site-packages/yum/misc.py", line 1165, in repo_gen_decompress return decompress(filename, dest=dest, check_timestamps=True) File "/usr/lib/python2.7/site-packages/yum/misc.py", line 1150, in decompress _decompress_chunked(filename, out, ztype) File "/usr/lib/python2.7/site-packages/yum/misc.py", line 791, in _decompress_chunked raise Errors.MiscError, msg yum.Errors.MiscError: Error reading from file /home/jenkins/mirrors_cache/epel-el6/9793d6512befee1760fec5220e44a82e73b95e31eeae1a943256a536a6046eb8-primary.xml.gz: Not a gzipped file Build step 'Execute shell' marked build as failure

1 1

[JIRA] (OVIRT-2354) Integrate global setup and cleanup in STDCI container
by Daniel Belenky (oVirt JIRA) 26 Jul '18

26 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2354?page=com.atlassian.jira.… ] Daniel Belenky updated OVIRT-2354: ---------------------------------- Status: In Progress (was: To Do) > Integrate global setup and cleanup in STDCI container > ----------------------------------------------------- > > Key: OVIRT-2354 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2354 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Gal Ben Haim > Assignee: infra > -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2148) 005_network_by_label.assign_hosts_network_label
by Dafna Ron (oVirt JIRA) 26 Jul '18

26 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2148?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2148: ---------------------------------- we had a failure with the same issue ovirt-ansible-infra on ovirt-4.2 http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2712/ it may be that the test in ost needs to be fixed because I am not seeing a problem with the network labels. there is a host that is killed as part of the recovery tests so it may be that this is the issue > 005_network_by_label.assign_hosts_network_label > ------------------------------------------------ > > Key: OVIRT-2148 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2148 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_race > > Change 91614,10 (ovirt-provider-ovn) is probably the reason behind recent > system test failures in the "ovirt-4.2" change queue and needs to be fixed. > This change had been removed from the testing queue. Artifacts build from this > change will not be released until it is fixed. > For further details about the change see: > https://gerrit.ovirt.org/#/c/91614/10 > For failed test results see: > http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2312/ > Fault reason is "Operation Failed". Fault detail is "[Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later.]". HTTP response code is 409. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2148) 005_network_by_label.assign_hosts_network_label
by Dafna Ron (oVirt JIRA) 26 Jul '18

26 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2148?page=com.atlassian.jira.… ] Dafna Ron reopened OVIRT-2148: ------------------------------ > 005_network_by_label.assign_hosts_network_label > ------------------------------------------------ > > Key: OVIRT-2148 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2148 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_race > > Change 91614,10 (ovirt-provider-ovn) is probably the reason behind recent > system test failures in the "ovirt-4.2" change queue and needs to be fixed. > This change had been removed from the testing queue. Artifacts build from this > change will not be released until it is fixed. > For further details about the change see: > https://gerrit.ovirt.org/#/c/91614/10 > For failed test results see: > http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2312/ > Fault reason is "Operation Failed". Fault detail is "[Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later.]". HTTP response code is 409. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[CQ]: 7e0b592 (ovirt-ansible-infra) failed "ovirt-4.2" system tests
by oVirt Jenkins 26 Jul '18

26 Jul '18

Change 7e0b592 (ovirt-ansible-infra) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://github.com/oVirt/ovirt-ansible-infra/commit/7e0b592fec19f48a0ad861e… For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2712/

1 0

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-4.2 - Build # 385 - Failure!
by jenkins＠jenkins.phx.ovirt.org 26 Jul '18

26 Jul '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/385/ Build Number: 385 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #385 [Gal Ben Haim] compat: Set "ex" bash flags ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.add_hosts Error Message: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML -------------------- >> begin captured logging << -------------------- ovirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7fbdb63105f0> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 254, in _host_is_up_4 host_obj = host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 36846, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 229, in add_hosts add_hosts_4(prefix) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 273, in add_hosts_4 testlib.assert_true_within(_host_is_up_4, timeout=15*60) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 263, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 254, in _host_is_up_4 host_obj = host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 36846, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type raise Error(msg) 'The response content type \'text/html; charset=iso-8859-1\' isn\'t the expected XML\n-------------------- >> begin captured logging << --------------------\novirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7fbdb63105f0>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 254, in _host_is_up_4\n host_obj = host_service.get()\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 36846, in get\n return self._internal_get(headers, query, wait)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get\n return future.wait() if wait else future\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait\n return self._code(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback\n self._check_fault(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault\n body = self._internal_read_body(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body\n self._connection.check_xml_content_type(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type\n response.headers\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type\n raise Error(msg)\nError: The response content type \'text/html; charset=iso-8859-1\' isn\'t the expected XML\n--------------------- >> end captured logging << ---------------------'

1 1

[JIRA] (OVIRT-2374) servers added to Jenkins with no sub-domain
by Barak Korren (oVirt JIRA) 26 Jul '18

26 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2374?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-2374: -------------------------------- Resolution: Won't Fix Status: Done (was: To Do) we'll remove these server soon anyway and convert them to run containers, so not going to fix this for now. > servers added to Jenkins with no sub-domain > ------------------------------------------- > > Key: OVIRT-2374 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2374 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > > the new servers added have a different dns sub domain then the old ones. > in Jenkins, the servers were added without the sub-domain > was that intentional? it would mean we would have to go to the configuration of the servers to know what sub-domain it has. unless there is a reason which we don't want the sub domains, can we add it to Jenkins? > 20:58:25 [upgrade-from-release-suit] Running on ovirt-srv07 in /home/jenkins/workspace/ovirt-master_change-queue-tester -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2374) servers added to Jenkins with no sub-domain
by Dafna Ron (oVirt JIRA) 26 Jul '18

26 Jul '18

Dafna Ron created OVIRT-2374: -------------------------------- Summary: servers added to Jenkins with no sub-domain Key: OVIRT-2374 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2374 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Dafna Ron Assignee: infra the new servers added have a different dns sub domain then the old ones. in Jenkins, the servers were added without the sub-domain was that intentional? it would mean we would have to go to the configuration of the servers to know what sub-domain it has. unless there is a reason which we don't want the sub domains, can we add it to Jenkins? 20:58:25 [upgrade-from-release-suit] Running on ovirt-srv07 in /home/jenkins/workspace/ovirt-master_change-queue-tester -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[CQ]: 92721,15 (ovirt-engine) failed "ovirt-master" system tests
by oVirt Jenkins 26 Jul '18

26 Jul '18

Change 92721,15 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/92721/15 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8793/

2 1

[oVirt Jenkins] ovirt-system-tests_compat-4.2-suite-master - Build # 3 - Failure!
by jenkins＠jenkins.phx.ovirt.org 26 Jul '18

26 Jul '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.2-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.2-suite-master/3/ Build Number: 3 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #3 [Lev Veyde] 004_basic_sanity.py cleanup ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.preview_snapshot_with_memory_42 Error Message: False != True after 600 seconds Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 78, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/004_basic_sanity.py", line 431, in preview_snapshot_with_memory pass File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ self.gen.next() File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test_utils/__init__.py", line 264, in TestEvent lambda: File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 271, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 258, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 237, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 600 seconds

1 9

[oVirt Jenkins] ovirt-appliance_master_build-artifacts-fc28-x86_64 - Build # 8 - Failure!
by jenkins＠jenkins.phx.ovirt.org 26 Jul '18

26 Jul '18

Project: http://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-fc28-x8… Build: http://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-fc28-x8… Build Number: 8 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #8 [Sandro Bonazzola] add kmajumde(a)redhat.com [Barak Korren] Refactor: consolidate STDCI reporting code [Barak Korren] Enable customizing GitHub report URL from DSL [Daniel Belenky] Temporary disable s390x jobs [Yuval Turgeman] automation: remove build-artifacts* from stdci-v2 ----------------- Failed Tests: ----------------- No tests ran.

1 1

[JIRA] (OVIRT-2373) Firefox claims 'gerrit.ovirt.org' uses weak encription
by Barak Korren (oVirt JIRA) 26 Jul '18

26 Jul '18

Barak Korren created OVIRT-2373: ----------------------------------- Summary: Firefox claims 'gerrit.ovirt.org' uses weak encription Key: OVIRT-2373 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2373 Project: oVirt - virtualization made easy Issue Type: New Feature Components: Gerrit/git Reporter: Barak Korren Assignee: infra -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2368) Please create some ovirt-ansible-* github repositories
by Barak Korren (oVirt JIRA) 26 Jul '18

26 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2368?page=com.atlassian.jira.… ] Barak Korren reassigned OVIRT-2368: ----------------------------------- Assignee: Barak Korren (was: infra) > Please create some ovirt-ansible-* github repositories > ------------------------------------------------------ > > Key: OVIRT-2368 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2368 > Project: oVirt - virtualization made easy > Issue Type: Task > Components: GitHub > Reporter: Lucie Leistnerova > Assignee: Barak Korren > > Please create this github repositories: > ovirt-ansible-dump-db > ovirt-ansible-rename-db > ovirt-ansible-remote-db > ovirt-ansible-local-to-remote-db > And assign leistnerova as administrator. > Thank you. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2372) Document the STDCI DSL 'release branches' option
by Barak Korren (oVirt JIRA) 26 Jul '18

26 Jul '18

Barak Korren created OVIRT-2372: ----------------------------------- Summary: Document the STDCI DSL 'release branches' option Key: OVIRT-2372 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2372 Project: oVirt - virtualization made easy Issue Type: New Feature Components: Change Queue, Documentation, STDCI DSL Reporter: Barak Korren Assignee: infra -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2371) Document the STDCI DSL 'reporting' option
by Barak Korren (oVirt JIRA) 26 Jul '18

26 Jul '18

Barak Korren created OVIRT-2371: ----------------------------------- Summary: Document the STDCI DSL 'reporting' option Key: OVIRT-2371 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2371 Project: oVirt - virtualization made easy Issue Type: New Feature Components: Documentation, STDCI DSL Reporter: Barak Korren Assignee: infra -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[oVirt Jenkins] ovirt-system-tests_he-basic-ansible-suite-4.2 - Build # 407 - Failure!
by jenkins＠jenkins.phx.ovirt.org 26 Jul '18

26 Jul '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4.2/… Build Number: 407 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #407 [Milan Zamazal] Wait for snapshot events correctly in preview_snapshot_with_memory [Gal Ben Haim] ost: Adding jobs for compat suite [Ehud Yonasi] Changed default oVirt ver to 4.2 for ost [Daniel Belenky] docker_cleanup: fix already removed container race [Yuval Turgeman] Add ovirt-node-ng-image project to stdci-v2 ----------------- Failed Tests: ----------------- All tests passed

1 1

oVirt infra daily report - unstable production jobs - 532
by jenkins＠jenkins.phx.ovirt.org 26 Jul '18

26 Jul '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/532//artifact/exported-a… Cheers, Jenkins

1 0

[JIRA] (OVIRT-2363) 008_basic_ui_sanity.start_grid failure with error "[Errno 2] No such file or directory"
by Barak Korren (oVirt JIRA) 25 Jul '18

25 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2363?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-2363: ------------------------------------- no, probably just need to erase mock cache on the hosts where this is happening to get a new one created with docker installed. > 008_basic_ui_sanity.start_grid failure with error "[Errno 2] No such file or directory" > --------------------------------------------------------------------------------------- > > Key: OVIRT-2363 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2363 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_failures > > patch that failed is: https://gerrit.ovirt.org/#/c/92882/ - core: hotplug new disk only if VM is up or paused states > I don't think the patch causes a regression but I am looking at the logs and checking to see if there was a behavior change which caused the ost test itself to fail. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[oVirt Jenkins] ovirt-system-tests_compat-4.1-suite-master - Build # 2 - Failure!
by jenkins＠jenkins.phx.ovirt.org 25 Jul '18

25 Jul '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.1-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.1-suite-master/2/ Build Number: 2 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #2 [Lev Veyde] 004_basic_sanity.py cleanup ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.preview_snapshot_with_memory_41 Error Message: False != True after 600 seconds Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 78, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/workspace/ovirt-system-tests_compat-4.1-suite-master/ovirt-system-tests/compat-4.1-suite-master/test-scenarios/004_basic_sanity.py", line 431, in preview_snapshot_with_memory pass File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ self.gen.next() File "/home/jenkins/workspace/ovirt-system-tests_compat-4.1-suite-master/ovirt-system-tests/compat-4.1-suite-master/test_utils/__init__.py", line 264, in TestEvent lambda: File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 271, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 258, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 237, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 600 seconds

1 8

[CQ]: 89641,14 (ovirt-engine) failed "ovirt-master" system tests
by oVirt Jenkins 25 Jul '18

25 Jul '18

Change 89641,14 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/89641/14 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8790/

1 0

oVirt infra weekly meeting notes 25.07.2018
by Evgheni Dereveanchin 25 Jul '18

25 Jul '18

Hi everyone, Please find the topics of this week's infra meeting below: *PHX: - Jenkins updated OVIRT-2262 <https://ovirt-jira.atlassian.net/browse/OVIRT-2262>- Let’s Encrypt SSL certificate installed on PHX engine OVIRT-531 <https://ovirt-jira.atlassian.net/browse/OVIRT-531>- 3 hosts rebuilt as bare metals OVIRT-2306 <https://ovirt-jira.atlassian.net/browse/OVIRT-2306>- 2 hosts upgraded to CentOS 7.5, rest on the way- Resources server crash reported OVIRT-2338 <https://ovirt-jira.atlassian.net/browse/OVIRT-2338>- Network flop between RDU and PHX caused monitoring alerts OVIRT-2356 <https://ovirt-jira.atlassian.net/browse/OVIRT-2356>- Patch sent to update PHX oVirt setup RTD page https://gerrit.ovirt.org/#/c/93166/ <https://gerrit.ovirt.org/#/c/93166/> Other topics: - Website moved to OSAS OVIRT-1799 <https://ovirt-jira.atlassian.net/browse/OVIRT-1799>- s390x slave down - builds for this arch temporarily disabledAI: - Shut down old website VM (ensure we’re monitoring the right one)- Document Let’s encrypt on PHX engine for DR purposes- Provide IRC nicks for direct contacts upstream* -- Regards, Evgheni Dereveanchin

1 0

[CQ]: 92924, 14 (ovirt-provider-ovn) failed "ovirt-master" system tests
by oVirt Jenkins 25 Jul '18

25 Jul '18

Change 92924,14 (ovirt-provider-ovn) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/92924/14 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8782/

2 1

[JIRA] (OVIRT-2363) 008_basic_ui_sanity.start_grid failure with error "[Errno 2] No such file or directory"
by Dafna Ron (oVirt JIRA) 25 Jul '18

25 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2363?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2363: ---------------------------------- third project to fail on this: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8782/ [~bkorren(a)redhat.com] [~ederevea] any chance that we have a new issue that is causing the same failures? > 008_basic_ui_sanity.start_grid failure with error "[Errno 2] No such file or directory" > --------------------------------------------------------------------------------------- > > Key: OVIRT-2363 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2363 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_failures > > patch that failed is: https://gerrit.ovirt.org/#/c/92882/ - core: hotplug new disk only if VM is up or paused states > I don't think the patch causes a regression but I am looking at the logs and checking to see if there was a behavior change which caused the ost test itself to fail. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-master - Build # 222 - Failure!
by jenkins＠jenkins.phx.ovirt.org 25 Jul '18

25 Jul '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/222/ Build Number: 222 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #222 [Milan Zamazal] Wait for snapshot events correctly in preview_snapshot_with_memory ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 008_restart_he_vm.restart_he_vm Error Message: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2518 (Wed Jul 25 04:20:06 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2519 (Wed Jul 25 04:20:06 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f3fa6dd6", "local_conf_timestamp": 2519, "host-ts": 2518}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2512 (Wed Jul 25 04:19:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2512 (Wed Jul 25 04:19:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "eng -------------------- >> begin captured logging << -------------------- lago.ssh: DEBUG: start task:932dbfdb-8f78-42c6-9b1e-6aadee5d8091:Get ssh client for lago-he-node-ng-suite-master-host-1: lago.ssh: DEBUG: end task:932dbfdb-8f78-42c6-9b1e-6aadee5d8091:Get ssh client for lago-he-node-ng-suite-master-host-1: lago.ssh: DEBUG: Running 7763a99a on lago-he-node-ng-suite-master-host-1: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 7763a99a on lago-he-node-ng-suite-master-host-1 returned with 0 lago.ssh: DEBUG: Command 7763a99a on lago-he-node-ng-suite-master-host-1 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2275 (Wed Jul 25 04:16:03 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2275 (Wed Jul 25 04:16:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "634493fe", "local_conf_timestamp": 2275, "host-ts": 2275}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2281 (Wed Jul 25 04:16:08 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2281 (Wed Jul 25 04:16:08 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69ca08d9", "local_conf_timestamp": 2281, "host-ts": 2281}, "global_maintenance": true} root: INFO: * Shutting down HE VM on host: lago-he-node-ng-suite-master-host-0 lago.ssh: DEBUG: start task:2bd52306-1957-43f5-a4cd-a16a4223bbd2:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:2bd52306-1957-43f5-a4cd-a16a4223bbd2:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 78525040 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-shutdown lago.ssh: DEBUG: Command 78525040 on lago-he-node-ng-suite-master-host-0 returned with 0 root: INFO: * Command succeeded root: INFO: * Waiting for VM to be down... lago.ssh: DEBUG: start task:75e52da9-c741-4089-8bda-708623dbb4cc:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:75e52da9-c741-4089-8bda-708623dbb4cc:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 797baba6 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 797baba6 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 797baba6 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2285 (Wed Jul 25 04:16:13 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2286 (Wed Jul 25 04:16:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "bc8dc98f", "local_conf_timestamp": 2286, "host-ts": 2285}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2281 (Wed Jul 25 04:16:08 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2281 (Wed Jul 25 04:16:08 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69ca08d9", "local_conf_timestamp": 2281, "host-ts": 2281}, "global_maintenance": true} lago.ssh: DEBUG: start task:ac00a93a-e808-4f72-bb6f-f29c7bc259a6:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:ac00a93a-e808-4f72-bb6f-f29c7bc259a6:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 812bb044 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 812bb044 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 812bb044 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2295 (Wed Jul 25 04:16:22 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2295 (Wed Jul 25 04:16:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf71ee59", "local_conf_timestamp": 2295, "host-ts": 2295}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2291 (Wed Jul 25 04:16:18 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2291 (Wed Jul 25 04:16:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "47e1cb10", "local_conf_timestamp": 2291, "host-ts": 2291}, "global_maintenance": true} lago.ssh: DEBUG: start task:bb5c3ebf-f315-4147-9664-83fbdfb0b5e8:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:bb5c3ebf-f315-4147-9664-83fbdfb0b5e8:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 83e14d9e on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 83e14d9e on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 83e14d9e on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2305 (Wed Jul 25 04:16:33 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2305 (Wed Jul 25 04:16:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "16c42d10", "local_conf_timestamp": 2305, "host-ts": 2305}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2301 (Wed Jul 25 04:16:28 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2301 (Wed Jul 25 04:16:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "21734107", "local_conf_timestamp": 2301, "host-ts": 2301}, "global_maintenance": true} root: INFO: * VM is down. root: INFO: * Stopping services... lago.ssh: DEBUG: start task:a91ce735-904b-4589-8d49-374d4d4e5233:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:a91ce735-904b-4589-8d49-374d4d4e5233:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 84cdbff8 on lago-he-node-ng-suite-master-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command 84cdbff8 on lago-he-node-ng-suite-master-host-0 returned with 0 root: INFO: * Starting services... lago.ssh: DEBUG: start task:69e2b63b-31fc-464e-b7ff-c499a95be86e:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:69e2b63b-31fc-464e-b7ff-c499a95be86e:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 8dac9a18 on lago-he-node-ng-suite-master-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command 8dac9a18 on lago-he-node-ng-suite-master-host-0 returned with 0 root: INFO: * Waiting for agent to be ready... lago.ssh: DEBUG: start task:5c41febb-2a25-4889-a7c1-5c9da7a33f96:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:5c41febb-2a25-4889-a7c1-5c9da7a33f96:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 909bffb6 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 909bffb6 on lago-he-node-ng-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 909bffb6 on lago-he-node-ng-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:56d0e0c8-3a30-4b6e-acf5-ba1e387ceb7c:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:56d0e0c8-3a30-4b6e-acf5-ba1e387ceb7c:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 977e0a9a on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 977e0a9a on lago-he-node-ng-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 977e0a9a on lago-he-node-ng-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:4b36c31c-c6b7-4bbb-b038-b67e2a26e788:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:4b36c31c-c6b7-4bbb-b038-b67e2a26e788:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 9a1513fc on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 9a1513fc on lago-he-node-ng-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 9a1513fc on lago-he-node-ng-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:e3e68036-c86b-4a3e-ad6d-788272025c42:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:e3e68036-c86b-4a3e-ad6d-788272025c42:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 9cbc9a94 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 9cbc9a94 on lago-he-node-ng-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 9cbc9a94 on lago-he-node-ng-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:7b9f8a19-b7dc-4d00-9202-e07d88888dfe:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:7b9f8a19-b7dc-4d00-9202-e07d88888dfe:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 9f7865c4 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 9f7865c4 on lago-he-node-ng-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 9f7865c4 on lago-he-node-ng-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:1944f900-1947-46ca-b701-3f71d3025a23:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:1944f900-1947-46ca-b701-3f71d3025a23:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running a212e07a on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command a212e07a on lago-he-node-ng-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command a212e07a on lago-he-node-ng-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:289afe1e-8a48-4e1f-b585-c7afe283cc2a:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:289afe1e-8a48-4e1f-b585-c7afe283cc2a:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running a4b92d66 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command a4b92d66 on lago-he-node-ng-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command a4b92d66 on lago-he-node-ng-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:65b385d4-2abe-4c8a-9f9f-37904c3320b5:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:65b385d4-2abe-4c8a-9f9f-37904c3320b5:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running a755a3f6 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command a755a3f6 on lago-he-node-ng-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command a755a3f6 on lago-he-node-ng-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:179389d5-7a94-4742-9328-73c41ed07ed0:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:179389d5-7a94-4742-9328-73c41ed07ed0:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running a9fe6f48 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command a9fe6f48 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command a9fe6f48 on lago-he-node-ng-suite-master-host-0 output: !! Cluster is in GLOBAL MAINTENANCE mode !! --== Host lago-he-node-ng-suite-master-host-0 (id: 1) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : lago-he-node-ng-suite-master-host-0 Host ID : 1 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"} Score : 0 stopped : False Local maintenance : False crc32 : 744e472a local_conf_timestamp : 2368 Host timestamp : 2367 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2367 (Wed Jul 25 04:17:35 2018) host-id=1 score=0 vm_conf_refresh_time=2368 (Wed Jul 25 04:17:35 2018) conf_on_shared_storage=True maintenance=False state=ReinitializeFSM stopped=False --== Host lago-he-node-ng-suite-master-host-1 (id: 2) status ==-- conf_on_shared_storage : True Status up-to-date : False Hostname : lago-he-node-ng-suite-master-host-1 Host ID : 2 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : c58bc9b1 local_conf_timestamp : 2361 Host timestamp : 2361 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2361 (Wed Jul 25 04:17:28 2018) host-id=2 score=3400 vm_conf_refresh_time=2361 (Wed Jul 25 04:17:29 2018) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False !! Cluster is in GLOBAL MAINTENANCE mode !! root: INFO: * Agent is ready. root: INFO: * Starting VM... lago.ssh: DEBUG: start task:771d13e3-aa83-4b0c-ad2f-73254bee44b5:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:771d13e3-aa83-4b0c-ad2f-73254bee44b5:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running ab02495a on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-start lago.ssh: DEBUG: Command ab02495a on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command ab02495a on lago-he-node-ng-suite-master-host-0 output: VM exists and is down, cleaning up and restarting root: INFO: * Command succeeded root: INFO: * Waiting for VM to be UP... lago.ssh: DEBUG: start task:9b23a87d-00d7-4b1b-9df3-d4f2a93a347a:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:9b23a87d-00d7-4b1b-9df3-d4f2a93a347a:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running ad1c960a on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ad1c960a on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command ad1c960a on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2367 (Wed Jul 25 04:17:35 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=2368 (Wed Jul 25 04:17:35 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=ReinitializeFSM\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "744e472a", "local_conf_timestamp": 2368, "host-ts": 2367}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2361 (Wed Jul 25 04:17:28 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2361 (Wed Jul 25 04:17:29 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c58bc9b1", "local_conf_timestamp": 2361, "host-ts": 2361}, "global_maintenance": true} lago.ssh: DEBUG: start task:38723e85-fc65-4a4c-903e-49754f691574:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:38723e85-fc65-4a4c-903e-49754f691574:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running b40283bc on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b40283bc on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command b40283bc on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2377 (Wed Jul 25 04:17:44 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2377 (Wed Jul 25 04:17:45 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "40391805", "local_conf_timestamp": 2377, "host-ts": 2377}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2381 (Wed Jul 25 04:17:48 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2381 (Wed Jul 25 04:17:49 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8b5c590e", "local_conf_timestamp": 2381, "host-ts": 2381}, "global_maintenance": true} lago.ssh: DEBUG: start task:cd1092f6-e8c6-40ee-99d2-6a6edd7be1b5:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:cd1092f6-e8c6-40ee-99d2-6a6edd7be1b5:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running b6c56286 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b6c56286 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command b6c56286 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2387 (Wed Jul 25 04:17:55 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2388 (Wed Jul 25 04:17:55 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d2ce9013", "local_conf_timestamp": 2388, "host-ts": 2387}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2381 (Wed Jul 25 04:17:48 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2381 (Wed Jul 25 04:17:49 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8b5c590e", "local_conf_timestamp": 2381, "host-ts": 2381}, "global_maintenance": true} root: INFO: * VM is UP. root: INFO: * Waiting for engine to start... lago.ssh: DEBUG: start task:567f9428-43b1-44f5-ad18-0f4a01c91b13:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:567f9428-43b1-44f5-ad18-0f4a01c91b13:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running b7c81b4c on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b7c81b4c on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command b7c81b4c on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2387 (Wed Jul 25 04:17:55 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2388 (Wed Jul 25 04:17:55 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d2ce9013", "local_conf_timestamp": 2388, "host-ts": 2387}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2381 (Wed Jul 25 04:17:48 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2381 (Wed Jul 25 04:17:49 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8b5c590e", "local_conf_timestamp": 2381, "host-ts": 2381}, "global_maintenance": true} lago.ssh: DEBUG: start task:029af7ab-0480-465a-ad7d-9d3a37b5b355:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:029af7ab-0480-465a-ad7d-9d3a37b5b355:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running bec6f814 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command bec6f814 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command bec6f814 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2398 (Wed Jul 25 04:18:05 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2398 (Wed Jul 25 04:18:06 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "412e4ba9", "local_conf_timestamp": 2398, "host-ts": 2398}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2391 (Wed Jul 25 04:17:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2392 (Wed Jul 25 04:17:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "3c7ecb7d", "local_conf_timestamp": 2392, "host-ts": 2391}, "global_maintenance": true} lago.ssh: DEBUG: start task:13363bf6-bedc-4e09-814b-bf03d9839892:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:13363bf6-bedc-4e09-814b-bf03d9839892:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running c1a71ce4 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command c1a71ce4 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command c1a71ce4 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2408 (Wed Jul 25 04:18:16 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2408 (Wed Jul 25 04:18:16 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9432151", "local_conf_timestamp": 2408, "host-ts": 2408}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2402 (Wed Jul 25 04:18:09 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2402 (Wed Jul 25 04:18:09 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9a202027", "local_conf_timestamp": 2402, "host-ts": 2402}, "global_maintenance": true} lago.ssh: DEBUG: start task:8e86ea8a-ed97-49c2-aa31-0d07662c480f:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:8e86ea8a-ed97-49c2-aa31-0d07662c480f:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running c4dffe62 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command c4dffe62 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command c4dffe62 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2408 (Wed Jul 25 04:18:16 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2408 (Wed Jul 25 04:18:16 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9432151", "local_conf_timestamp": 2408, "host-ts": 2408}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2402 (Wed Jul 25 04:18:09 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2402 (Wed Jul 25 04:18:09 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9a202027", "local_conf_timestamp": 2402, "host-ts": 2402}, "global_maintenance": true} lago.ssh: DEBUG: start task:e801a931-3f5c-463c-b00e-07905169e39b:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:e801a931-3f5c-463c-b00e-07905169e39b:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running c86e9048 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command c86e9048 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command c86e9048 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2418 (Wed Jul 25 04:18:26 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2419 (Wed Jul 25 04:18:26 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "625c3f40", "local_conf_timestamp": 2419, "host-ts": 2418}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2412 (Wed Jul 25 04:18:20 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2413 (Wed Jul 25 04:18:20 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c30cd378", "local_conf_timestamp": 2413, "host-ts": 2412}, "global_maintenance": true} lago.ssh: DEBUG: start task:fc9d9c05-cc70-49f4-9195-5801981130da:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:fc9d9c05-cc70-49f4-9195-5801981130da:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running cb60cc6c on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command cb60cc6c on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command cb60cc6c on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2418 (Wed Jul 25 04:18:26 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2419 (Wed Jul 25 04:18:26 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "625c3f40", "local_conf_timestamp": 2419, "host-ts": 2418}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2412 (Wed Jul 25 04:18:20 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2413 (Wed Jul 25 04:18:20 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c30cd378", "local_conf_timestamp": 2413, "host-ts": 2412}, "global_maintenance": true} lago.ssh: DEBUG: start task:a08eb28b-2662-4ab3-a80b-77ef2581a073:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:a08eb28b-2662-4ab3-a80b-77ef2581a073:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running cee93edc on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command cee93edc on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command cee93edc on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2428 (Wed Jul 25 04:18:35 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2428 (Wed Jul 25 04:18:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ace23939", "local_conf_timestamp": 2428, "host-ts": 2428}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2422 (Wed Jul 25 04:18:29 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2422 (Wed Jul 25 04:18:29 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c677a7b5", "local_conf_timestamp": 2422, "host-ts": 2422}, "global_maintenance": true} lago.ssh: DEBUG: start task:5475c36c-7873-4bb4-b0a4-89b7a176ed10:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:5475c36c-7873-4bb4-b0a4-89b7a176ed10:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running d2517044 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command d2517044 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command d2517044 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2438 (Wed Jul 25 04:18:46 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2439 (Wed Jul 25 04:18:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b02c5c74", "local_conf_timestamp": 2439, "host-ts": 2438}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2432 (Wed Jul 25 04:18:39 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2432 (Wed Jul 25 04:18:40 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e85c647c", "local_conf_timestamp": 2432, "host-ts": 2432}, "global_maintenance": true} lago.ssh: DEBUG: start task:378598d4-4368-44ff-b912-b640c5a049b0:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:378598d4-4368-44ff-b912-b640c5a049b0:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running d56c33a4 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command d56c33a4 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command d56c33a4 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2438 (Wed Jul 25 04:18:46 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2439 (Wed Jul 25 04:18:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b02c5c74", "local_conf_timestamp": 2439, "host-ts": 2438}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2432 (Wed Jul 25 04:18:39 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2432 (Wed Jul 25 04:18:40 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e85c647c", "local_conf_timestamp": 2432, "host-ts": 2432}, "global_maintenance": true} lago.ssh: DEBUG: start task:7f5db09d-6b15-4582-8818-3c99d0291ce5:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:7f5db09d-6b15-4582-8818-3c99d0291ce5:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running d863856c on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command d863856c on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command d863856c on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2448 (Wed Jul 25 04:18:55 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2448 (Wed Jul 25 04:18:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "0f24c3a2", "local_conf_timestamp": 2448, "host-ts": 2448}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2442 (Wed Jul 25 04:18:49 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2442 (Wed Jul 25 04:18:49 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "228f2f03", "local_conf_timestamp": 2442, "host-ts": 2442}, "global_maintenance": true} lago.ssh: DEBUG: start task:639271a3-0803-4020-966c-200e8adb5117:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:639271a3-0803-4020-966c-200e8adb5117:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running db6b7cf6 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command db6b7cf6 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command db6b7cf6 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2448 (Wed Jul 25 04:18:55 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2448 (Wed Jul 25 04:18:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "0f24c3a2", "local_conf_timestamp": 2448, "host-ts": 2448}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2442 (Wed Jul 25 04:18:49 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2442 (Wed Jul 25 04:18:49 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "228f2f03", "local_conf_timestamp": 2442, "host-ts": 2442}, "global_maintenance": true} lago.ssh: DEBUG: start task:d1c10391-bcf5-4501-ae6f-47a95d795283:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:d1c10391-bcf5-4501-ae6f-47a95d795283:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running de65a36e on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command de65a36e on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command de65a36e on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2458 (Wed Jul 25 04:19:06 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2458 (Wed Jul 25 04:19:06 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "64ed9679", "local_conf_timestamp": 2458, "host-ts": 2458}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2452 (Wed Jul 25 04:18:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2452 (Wed Jul 25 04:18:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "0ca4ecca", "local_conf_timestamp": 2452, "host-ts": 2452}, "global_maintenance": true} lago.ssh: DEBUG: start task:4e03db00-d1d4-43b0-bcfd-a21d3c32b7a7:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:4e03db00-d1d4-43b0-bcfd-a21d3c32b7a7:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running e173ed40 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e173ed40 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command e173ed40 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2458 (Wed Jul 25 04:19:06 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2458 (Wed Jul 25 04:19:06 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "64ed9679", "local_conf_timestamp": 2458, "host-ts": 2458}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2452 (Wed Jul 25 04:18:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2452 (Wed Jul 25 04:18:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "0ca4ecca", "local_conf_timestamp": 2452, "host-ts": 2452}, "global_maintenance": true} lago.ssh: DEBUG: start task:d8173e8d-c1e7-49d4-8863-b1007ff2704f:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:d8173e8d-c1e7-49d4-8863-b1007ff2704f:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running e462c878 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e462c878 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command e462c878 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2468 (Wed Jul 25 04:19:15 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2468 (Wed Jul 25 04:19:15 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d8b66814", "local_conf_timestamp": 2468, "host-ts": 2468}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2462 (Wed Jul 25 04:19:10 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2463 (Wed Jul 25 04:19:10 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "09df9807", "local_conf_timestamp": 2463, "host-ts": 2462}, "global_maintenance": true} lago.ssh: DEBUG: start task:27e25ae4-3414-4640-8869-9cffad8f6814:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:27e25ae4-3414-4640-8869-9cffad8f6814:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running e794bfec on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e794bfec on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command e794bfec on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2468 (Wed Jul 25 04:19:15 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2468 (Wed Jul 25 04:19:15 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d8b66814", "local_conf_timestamp": 2468, "host-ts": 2468}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2462 (Wed Jul 25 04:19:10 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2463 (Wed Jul 25 04:19:10 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "09df9807", "local_conf_timestamp": 2463, "host-ts": 2462}, "global_maintenance": true} lago.ssh: DEBUG: start task:d23aa18c-4462-4239-8db5-c29feecac71e:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:d23aa18c-4462-4239-8db5-c29feecac71e:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running eb998bd6 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command eb998bd6 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command eb998bd6 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2478 (Wed Jul 25 04:19:26 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2479 (Wed Jul 25 04:19:26 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "41565544", "local_conf_timestamp": 2479, "host-ts": 2478}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2472 (Wed Jul 25 04:19:19 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2472 (Wed Jul 25 04:19:20 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "50f36b58", "local_conf_timestamp": 2472, "host-ts": 2472}, "global_maintenance": true} lago.ssh: DEBUG: start task:37e54f8a-9fe3-47cd-8896-bb560cc5e4d1:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:37e54f8a-9fe3-47cd-8896-bb560cc5e4d1:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running eed21502 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command eed21502 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command eed21502 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2478 (Wed Jul 25 04:19:26 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2479 (Wed Jul 25 04:19:26 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "41565544", "local_conf_timestamp": 2479, "host-ts": 2478}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2472 (Wed Jul 25 04:19:19 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2472 (Wed Jul 25 04:19:20 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "50f36b58", "local_conf_timestamp": 2472, "host-ts": 2472}, "global_maintenance": true} lago.ssh: DEBUG: start task:b6793d51-1124-40aa-9f42-a2107d7649cc:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:b6793d51-1124-40aa-9f42-a2107d7649cc:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running f25fec4e on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f25fec4e on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command f25fec4e on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2488 (Wed Jul 25 04:19:35 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2488 (Wed Jul 25 04:19:35 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "16f668c8", "local_conf_timestamp": 2488, "host-ts": 2488}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2482 (Wed Jul 25 04:19:30 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2483 (Wed Jul 25 04:19:30 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "470808b8", "local_conf_timestamp": 2483, "host-ts": 2482}, "global_maintenance": true} lago.ssh: DEBUG: start task:6fb40f9a-5636-47d5-a5e7-cf1835987996:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:6fb40f9a-5636-47d5-a5e7-cf1835987996:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running f5dc5560 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f5dc5560 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command f5dc5560 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2498 (Wed Jul 25 04:19:46 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2499 (Wed Jul 25 04:19:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "0a380d85", "local_conf_timestamp": 2499, "host-ts": 2498}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2492 (Wed Jul 25 04:19:39 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2492 (Wed Jul 25 04:19:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1e24fbe7", "local_conf_timestamp": 2492, "host-ts": 2492}, "global_maintenance": true} lago.ssh: DEBUG: start task:54953ae8-9f55-4cb1-bb4c-7ec68dc9879f:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:54953ae8-9f55-4cb1-bb4c-7ec68dc9879f:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running f99d7d32 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f99d7d32 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command f99d7d32 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2498 (Wed Jul 25 04:19:46 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2499 (Wed Jul 25 04:19:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "0a380d85", "local_conf_timestamp": 2499, "host-ts": 2498}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2492 (Wed Jul 25 04:19:39 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2492 (Wed Jul 25 04:19:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1e24fbe7", "local_conf_timestamp": 2492, "host-ts": 2492}, "global_maintenance": true} lago.ssh: DEBUG: start task:2d207ff4-7778-45e7-abe3-56af39bec1c8:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:2d207ff4-7778-45e7-abe3-56af39bec1c8:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running fd26f32a on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command fd26f32a on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command fd26f32a on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2508 (Wed Jul 25 04:19:55 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2508 (Wed Jul 25 04:19:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ef34089b", "local_conf_timestamp": 2508, "host-ts": 2508}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2502 (Wed Jul 25 04:19:49 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2502 (Wed Jul 25 04:19:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "78b671f0", "local_conf_timestamp": 2502, "host-ts": 2502}, "global_maintenance": true} lago.ssh: DEBUG: start task:4819a00d-a93c-4b9d-9439-77204d723a0a:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:4819a00d-a93c-4b9d-9439-77204d723a0a:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 00ce0cca on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 00ce0cca on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 00ce0cca on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2508 (Wed Jul 25 04:19:55 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2508 (Wed Jul 25 04:19:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ef34089b", "local_conf_timestamp": 2508, "host-ts": 2508}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2502 (Wed Jul 25 04:19:49 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2502 (Wed Jul 25 04:19:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "78b671f0", "local_conf_timestamp": 2502, "host-ts": 2502}, "global_maintenance": true} lago.ssh: DEBUG: start task:2eadab93-166a-4267-8a57-0d457b253235:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:2eadab93-166a-4267-8a57-0d457b253235:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 04474b64 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 04474b64 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 04474b64 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2518 (Wed Jul 25 04:20:06 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2519 (Wed Jul 25 04:20:06 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f3fa6dd6", "local_conf_timestamp": 2519, "host-ts": 2518}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2512 (Wed Jul 25 04:19:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2512 (Wed Jul 25 04:19:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "569db239", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true} lago.ssh: DEBUG: start task:697a24f7-bb86-4e63-abc3-0ee588786d0a:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:697a24f7-bb86-4e63-abc3-0ee588786d0a:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 07152f64 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 07152f64 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 07152f64 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2518 (Wed Jul 25 04:20:06 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2519 (Wed Jul 25 04:20:06 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f3fa6dd6", "local_conf_timestamp": 2519, "host-ts": 2518}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2512 (Wed Jul 25 04:19:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2512 (Wed Jul 25 04:19:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "eng ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f2f900dd140> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/008_restart_he_vm.py", line 165, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/008_restart_he_vm.py", line 106, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) RuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2518 (Wed Jul 25 04:20:06 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2519 (Wed Jul 25 04:20:06 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f3fa6dd6", "local_conf_timestamp": 2519, "host-ts": 2518}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2512 (Wed Jul 25 04:19:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2512 (Wed Jul 25 04:19:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "eng --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/008_restart_he_vm.py", line 49, in restart_he_vm _wait_for_engine_health(host) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/008_restart_he_vm.py", line 163, in _wait_for_engine_health testlib.assert_true_within_long(lambda: any( File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 271, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 258, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/008_restart_he_vm.py", line 165, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/008_restart_he_vm.py", line 106, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) 'could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2518 (Wed Jul 25 04:20:06 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2519 (Wed Jul 25 04:20:06 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f3fa6dd6", "local_conf_timestamp": 2519, "host-ts": 2518}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2512 (Wed Jul 25 04:19:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2512 (Wed Jul 25 04:19:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "eng\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:932dbfdb-8f78-42c6-9b1e-6aadee5d8091:Get ssh client for lago-he-node-ng-suite-master-host-1:\nlago.ssh: DEBUG: end task:932dbfdb-8f78-42c6-9b1e-6aadee5d8091:Get ssh client for lago-he-node-ng-suite-master-host-1:\nlago.ssh: DEBUG: Running 7763a99a on lago-he-node-ng-suite-master-host-1: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 7763a99a on lago-he-node-ng-suite-master-host-1 returned with 0\nlago.ssh: DEBUG: Command 7763a99a on lago-he-node-ng-suite-master-host-1 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2275 (Wed Jul 25 04:16:03 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2275 (Wed Jul 25 04:16:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "634493fe", "local_conf_timestamp": 2275, "host-ts": 2275}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2281 (Wed Jul 25 04:16:08 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2281 (Wed Jul 25 04:16:08 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69ca08d9", "local_conf_timestamp": 2281, "host-ts": 2281}, "global_maintenance": true}\n\nroot: INFO: * Shutting down HE VM on host: lago-he-node-ng-suite-master-host-0\nlago.ssh: DEBUG: start task:2bd52306-1957-43f5-a4cd-a16a4223bbd2:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:2bd52306-1957-43f5-a4cd-a16a4223bbd2:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 78525040 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-shutdown\nlago.ssh: DEBUG: Command 78525040 on lago-he-node-ng-suite-master-host-0 returned with 0\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be down...\nlago.ssh: DEBUG: start task:75e52da9-c741-4089-8bda-708623dbb4cc:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:75e52da9-c741-4089-8bda-708623dbb4cc:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 797baba6 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 797baba6 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 797baba6 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2285 (Wed Jul 25 04:16:13 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2286 (Wed Jul 25 04:16:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "bc8dc98f", "local_conf_timestamp": 2286, "host-ts": 2285}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2281 (Wed Jul 25 04:16:08 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2281 (Wed Jul 25 04:16:08 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69ca08d9", "local_conf_timestamp": 2281, "host-ts": 2281}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ac00a93a-e808-4f72-bb6f-f29c7bc259a6:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:ac00a93a-e808-4f72-bb6f-f29c7bc259a6:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 812bb044 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 812bb044 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 812bb044 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2295 (Wed Jul 25 04:16:22 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2295 (Wed Jul 25 04:16:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf71ee59", "local_conf_timestamp": 2295, "host-ts": 2295}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2291 (Wed Jul 25 04:16:18 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2291 (Wed Jul 25 04:16:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "47e1cb10", "local_conf_timestamp": 2291, "host-ts": 2291}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:bb5c3ebf-f315-4147-9664-83fbdfb0b5e8:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:bb5c3ebf-f315-4147-9664-83fbdfb0b5e8:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 83e14d9e on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 83e14d9e on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 83e14d9e on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2305 (Wed Jul 25 04:16:33 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2305 (Wed Jul 25 04:16:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "16c42d10", "local_conf_timestamp": 2305, "host-ts": 2305}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2301 (Wed Jul 25 04:16:28 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2301 (Wed Jul 25 04:16:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "21734107", "local_conf_timestamp": 2301, "host-ts": 2301}, "global_maintenance": true}\n\nroot: INFO: * VM is down.\nroot: INFO: * Stopping services...\nlago.ssh: DEBUG: start task:a91ce735-904b-4589-8d49-374d4d4e5233:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:a91ce735-904b-4589-8d49-374d4d4e5233:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 84cdbff8 on lago-he-node-ng-suite-master-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command 84cdbff8 on lago-he-node-ng-suite-master-host-0 returned with 0\nroot: INFO: * Starting services...\nlago.ssh: DEBUG: start task:69e2b63b-31fc-464e-b7ff-c499a95be86e:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:69e2b63b-31fc-464e-b7ff-c499a95be86e:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 8dac9a18 on lago-he-node-ng-suite-master-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command 8dac9a18 on lago-he-node-ng-suite-master-host-0 returned with 0\nroot: INFO: * Waiting for agent to be ready...\nlago.ssh: DEBUG: start task:5c41febb-2a25-4889-a7c1-5c9da7a33f96:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:5c41febb-2a25-4889-a7c1-5c9da7a33f96:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 909bffb6 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 909bffb6 on lago-he-node-ng-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 909bffb6 on lago-he-node-ng-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:56d0e0c8-3a30-4b6e-acf5-ba1e387ceb7c:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:56d0e0c8-3a30-4b6e-acf5-ba1e387ceb7c:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 977e0a9a on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 977e0a9a on lago-he-node-ng-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 977e0a9a on lago-he-node-ng-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:4b36c31c-c6b7-4bbb-b038-b67e2a26e788:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:4b36c31c-c6b7-4bbb-b038-b67e2a26e788:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 9a1513fc on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 9a1513fc on lago-he-node-ng-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 9a1513fc on lago-he-node-ng-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:e3e68036-c86b-4a3e-ad6d-788272025c42:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:e3e68036-c86b-4a3e-ad6d-788272025c42:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 9cbc9a94 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 9cbc9a94 on lago-he-node-ng-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 9cbc9a94 on lago-he-node-ng-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:7b9f8a19-b7dc-4d00-9202-e07d88888dfe:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:7b9f8a19-b7dc-4d00-9202-e07d88888dfe:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 9f7865c4 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 9f7865c4 on lago-he-node-ng-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 9f7865c4 on lago-he-node-ng-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:1944f900-1947-46ca-b701-3f71d3025a23:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:1944f900-1947-46ca-b701-3f71d3025a23:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running a212e07a on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command a212e07a on lago-he-node-ng-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command a212e07a on lago-he-node-ng-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:289afe1e-8a48-4e1f-b585-c7afe283cc2a:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:289afe1e-8a48-4e1f-b585-c7afe283cc2a:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running a4b92d66 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command a4b92d66 on lago-he-node-ng-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command a4b92d66 on lago-he-node-ng-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:65b385d4-2abe-4c8a-9f9f-37904c3320b5:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:65b385d4-2abe-4c8a-9f9f-37904c3320b5:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running a755a3f6 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command a755a3f6 on lago-he-node-ng-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command a755a3f6 on lago-he-node-ng-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:179389d5-7a94-4742-9328-73c41ed07ed0:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:179389d5-7a94-4742-9328-73c41ed07ed0:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running a9fe6f48 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command a9fe6f48 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a9fe6f48 on lago-he-node-ng-suite-master-host-0 output:\n \n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\n\n--== Host lago-he-node-ng-suite-master-host-0 (id: 1) status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : True\nHostname : lago-he-node-ng-suite-master-host-0\nHost ID : 1\nEngine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}\nScore : 0\nstopped : False\nLocal maintenance : False\ncrc32 : 744e472a\nlocal_conf_timestamp : 2368\nHost timestamp : 2367\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2367 (Wed Jul 25 04:17:35 2018)\n\thost-id=1\n\tscore=0\n\tvm_conf_refresh_time=2368 (Wed Jul 25 04:17:35 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=ReinitializeFSM\n\tstopped=False\n\n\n--== Host lago-he-node-ng-suite-master-host-1 (id: 2) status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : False\nHostname : lago-he-node-ng-suite-master-host-1\nHost ID : 2\nEngine status : unknown stale-data\nScore : 3400\nstopped : False\nLocal maintenance : False\ncrc32 : c58bc9b1\nlocal_conf_timestamp : 2361\nHost timestamp : 2361\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2361 (Wed Jul 25 04:17:28 2018)\n\thost-id=2\n\tscore=3400\n\tvm_conf_refresh_time=2361 (Wed Jul 25 04:17:29 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=GlobalMaintenance\n\tstopped=False\n\n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\nroot: INFO: * Agent is ready.\nroot: INFO: * Starting VM...\nlago.ssh: DEBUG: start task:771d13e3-aa83-4b0c-ad2f-73254bee44b5:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:771d13e3-aa83-4b0c-ad2f-73254bee44b5:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running ab02495a on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-start\nlago.ssh: DEBUG: Command ab02495a on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ab02495a on lago-he-node-ng-suite-master-host-0 output:\n VM exists and is down, cleaning up and restarting\n\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be UP...\nlago.ssh: DEBUG: start task:9b23a87d-00d7-4b1b-9df3-d4f2a93a347a:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:9b23a87d-00d7-4b1b-9df3-d4f2a93a347a:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running ad1c960a on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ad1c960a on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ad1c960a on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2367 (Wed Jul 25 04:17:35 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=2368 (Wed Jul 25 04:17:35 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=ReinitializeFSM\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "744e472a", "local_conf_timestamp": 2368, "host-ts": 2367}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2361 (Wed Jul 25 04:17:28 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2361 (Wed Jul 25 04:17:29 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c58bc9b1", "local_conf_timestamp": 2361, "host-ts": 2361}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:38723e85-fc65-4a4c-903e-49754f691574:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:38723e85-fc65-4a4c-903e-49754f691574:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running b40283bc on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b40283bc on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command b40283bc on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2377 (Wed Jul 25 04:17:44 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2377 (Wed Jul 25 04:17:45 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "40391805", "local_conf_timestamp": 2377, "host-ts": 2377}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2381 (Wed Jul 25 04:17:48 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2381 (Wed Jul 25 04:17:49 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8b5c590e", "local_conf_timestamp": 2381, "host-ts": 2381}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:cd1092f6-e8c6-40ee-99d2-6a6edd7be1b5:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:cd1092f6-e8c6-40ee-99d2-6a6edd7be1b5:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running b6c56286 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b6c56286 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command b6c56286 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2387 (Wed Jul 25 04:17:55 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2388 (Wed Jul 25 04:17:55 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d2ce9013", "local_conf_timestamp": 2388, "host-ts": 2387}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2381 (Wed Jul 25 04:17:48 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2381 (Wed Jul 25 04:17:49 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8b5c590e", "local_conf_timestamp": 2381, "host-ts": 2381}, "global_maintenance": true}\n\nroot: INFO: * VM is UP.\nroot: INFO: * Waiting for engine to start...\nlago.ssh: DEBUG: start task:567f9428-43b1-44f5-ad18-0f4a01c91b13:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:567f9428-43b1-44f5-ad18-0f4a01c91b13:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running b7c81b4c on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b7c81b4c on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command b7c81b4c on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2387 (Wed Jul 25 04:17:55 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2388 (Wed Jul 25 04:17:55 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d2ce9013", "local_conf_timestamp": 2388, "host-ts": 2387}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2381 (Wed Jul 25 04:17:48 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2381 (Wed Jul 25 04:17:49 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8b5c590e", "local_conf_timestamp": 2381, "host-ts": 2381}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:029af7ab-0480-465a-ad7d-9d3a37b5b355:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:029af7ab-0480-465a-ad7d-9d3a37b5b355:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running bec6f814 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command bec6f814 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command bec6f814 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2398 (Wed Jul 25 04:18:05 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2398 (Wed Jul 25 04:18:06 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "412e4ba9", "local_conf_timestamp": 2398, "host-ts": 2398}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2391 (Wed Jul 25 04:17:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2392 (Wed Jul 25 04:17:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "3c7ecb7d", "local_conf_timestamp": 2392, "host-ts": 2391}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:13363bf6-bedc-4e09-814b-bf03d9839892:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:13363bf6-bedc-4e09-814b-bf03d9839892:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running c1a71ce4 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command c1a71ce4 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command c1a71ce4 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2408 (Wed Jul 25 04:18:16 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2408 (Wed Jul 25 04:18:16 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9432151", "local_conf_timestamp": 2408, "host-ts": 2408}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2402 (Wed Jul 25 04:18:09 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2402 (Wed Jul 25 04:18:09 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9a202027", "local_conf_timestamp": 2402, "host-ts": 2402}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8e86ea8a-ed97-49c2-aa31-0d07662c480f:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:8e86ea8a-ed97-49c2-aa31-0d07662c480f:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running c4dffe62 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command c4dffe62 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command c4dffe62 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2408 (Wed Jul 25 04:18:16 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2408 (Wed Jul 25 04:18:16 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9432151", "local_conf_timestamp": 2408, "host-ts": 2408}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2402 (Wed Jul 25 04:18:09 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2402 (Wed Jul 25 04:18:09 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9a202027", "local_conf_timestamp": 2402, "host-ts": 2402}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:e801a931-3f5c-463c-b00e-07905169e39b:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:e801a931-3f5c-463c-b00e-07905169e39b:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running c86e9048 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command c86e9048 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command c86e9048 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2418 (Wed Jul 25 04:18:26 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2419 (Wed Jul 25 04:18:26 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "625c3f40", "local_conf_timestamp": 2419, "host-ts": 2418}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2412 (Wed Jul 25 04:18:20 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2413 (Wed Jul 25 04:18:20 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c30cd378", "local_conf_timestamp": 2413, "host-ts": 2412}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:fc9d9c05-cc70-49f4-9195-5801981130da:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:fc9d9c05-cc70-49f4-9195-5801981130da:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running cb60cc6c on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command cb60cc6c on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command cb60cc6c on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2418 (Wed Jul 25 04:18:26 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2419 (Wed Jul 25 04:18:26 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "625c3f40", "local_conf_timestamp": 2419, "host-ts": 2418}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2412 (Wed Jul 25 04:18:20 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2413 (Wed Jul 25 04:18:20 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c30cd378", "local_conf_timestamp": 2413, "host-ts": 2412}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:a08eb28b-2662-4ab3-a80b-77ef2581a073:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:a08eb28b-2662-4ab3-a80b-77ef2581a073:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running cee93edc on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command cee93edc on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command cee93edc on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2428 (Wed Jul 25 04:18:35 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2428 (Wed Jul 25 04:18:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ace23939", "local_conf_timestamp": 2428, "host-ts": 2428}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2422 (Wed Jul 25 04:18:29 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2422 (Wed Jul 25 04:18:29 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c677a7b5", "local_conf_timestamp": 2422, "host-ts": 2422}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:5475c36c-7873-4bb4-b0a4-89b7a176ed10:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:5475c36c-7873-4bb4-b0a4-89b7a176ed10:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running d2517044 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command d2517044 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command d2517044 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2438 (Wed Jul 25 04:18:46 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2439 (Wed Jul 25 04:18:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b02c5c74", "local_conf_timestamp": 2439, "host-ts": 2438}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2432 (Wed Jul 25 04:18:39 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2432 (Wed Jul 25 04:18:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e85c647c", "local_conf_timestamp": 2432, "host-ts": 2432}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:378598d4-4368-44ff-b912-b640c5a049b0:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:378598d4-4368-44ff-b912-b640c5a049b0:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running d56c33a4 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command d56c33a4 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command d56c33a4 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2438 (Wed Jul 25 04:18:46 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2439 (Wed Jul 25 04:18:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b02c5c74", "local_conf_timestamp": 2439, "host-ts": 2438}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2432 (Wed Jul 25 04:18:39 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2432 (Wed Jul 25 04:18:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e85c647c", "local_conf_timestamp": 2432, "host-ts": 2432}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:7f5db09d-6b15-4582-8818-3c99d0291ce5:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:7f5db09d-6b15-4582-8818-3c99d0291ce5:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running d863856c on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command d863856c on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command d863856c on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2448 (Wed Jul 25 04:18:55 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2448 (Wed Jul 25 04:18:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "0f24c3a2", "local_conf_timestamp": 2448, "host-ts": 2448}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2442 (Wed Jul 25 04:18:49 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2442 (Wed Jul 25 04:18:49 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "228f2f03", "local_conf_timestamp": 2442, "host-ts": 2442}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:639271a3-0803-4020-966c-200e8adb5117:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:639271a3-0803-4020-966c-200e8adb5117:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running db6b7cf6 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command db6b7cf6 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command db6b7cf6 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2448 (Wed Jul 25 04:18:55 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2448 (Wed Jul 25 04:18:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "0f24c3a2", "local_conf_timestamp": 2448, "host-ts": 2448}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2442 (Wed Jul 25 04:18:49 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2442 (Wed Jul 25 04:18:49 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "228f2f03", "local_conf_timestamp": 2442, "host-ts": 2442}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:d1c10391-bcf5-4501-ae6f-47a95d795283:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:d1c10391-bcf5-4501-ae6f-47a95d795283:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running de65a36e on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command de65a36e on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command de65a36e on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2458 (Wed Jul 25 04:19:06 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2458 (Wed Jul 25 04:19:06 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "64ed9679", "local_conf_timestamp": 2458, "host-ts": 2458}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2452 (Wed Jul 25 04:18:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2452 (Wed Jul 25 04:18:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "0ca4ecca", "local_conf_timestamp": 2452, "host-ts": 2452}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:4e03db00-d1d4-43b0-bcfd-a21d3c32b7a7:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:4e03db00-d1d4-43b0-bcfd-a21d3c32b7a7:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running e173ed40 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e173ed40 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command e173ed40 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2458 (Wed Jul 25 04:19:06 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2458 (Wed Jul 25 04:19:06 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "64ed9679", "local_conf_timestamp": 2458, "host-ts": 2458}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2452 (Wed Jul 25 04:18:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2452 (Wed Jul 25 04:18:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "0ca4ecca", "local_conf_timestamp": 2452, "host-ts": 2452}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:d8173e8d-c1e7-49d4-8863-b1007ff2704f:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:d8173e8d-c1e7-49d4-8863-b1007ff2704f:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running e462c878 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e462c878 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command e462c878 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2468 (Wed Jul 25 04:19:15 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2468 (Wed Jul 25 04:19:15 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d8b66814", "local_conf_timestamp": 2468, "host-ts": 2468}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2462 (Wed Jul 25 04:19:10 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2463 (Wed Jul 25 04:19:10 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "09df9807", "local_conf_timestamp": 2463, "host-ts": 2462}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:27e25ae4-3414-4640-8869-9cffad8f6814:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:27e25ae4-3414-4640-8869-9cffad8f6814:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running e794bfec on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e794bfec on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command e794bfec on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2468 (Wed Jul 25 04:19:15 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2468 (Wed Jul 25 04:19:15 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d8b66814", "local_conf_timestamp": 2468, "host-ts": 2468}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2462 (Wed Jul 25 04:19:10 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2463 (Wed Jul 25 04:19:10 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "09df9807", "local_conf_timestamp": 2463, "host-ts": 2462}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:d23aa18c-4462-4239-8db5-c29feecac71e:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:d23aa18c-4462-4239-8db5-c29feecac71e:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running eb998bd6 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command eb998bd6 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command eb998bd6 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2478 (Wed Jul 25 04:19:26 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2479 (Wed Jul 25 04:19:26 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "41565544", "local_conf_timestamp": 2479, "host-ts": 2478}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2472 (Wed Jul 25 04:19:19 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2472 (Wed Jul 25 04:19:20 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "50f36b58", "local_conf_timestamp": 2472, "host-ts": 2472}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:37e54f8a-9fe3-47cd-8896-bb560cc5e4d1:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:37e54f8a-9fe3-47cd-8896-bb560cc5e4d1:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running eed21502 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command eed21502 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command eed21502 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2478 (Wed Jul 25 04:19:26 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2479 (Wed Jul 25 04:19:26 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "41565544", "local_conf_timestamp": 2479, "host-ts": 2478}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2472 (Wed Jul 25 04:19:19 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2472 (Wed Jul 25 04:19:20 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "50f36b58", "local_conf_timestamp": 2472, "host-ts": 2472}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:b6793d51-1124-40aa-9f42-a2107d7649cc:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:b6793d51-1124-40aa-9f42-a2107d7649cc:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running f25fec4e on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f25fec4e on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command f25fec4e on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2488 (Wed Jul 25 04:19:35 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2488 (Wed Jul 25 04:19:35 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "16f668c8", "local_conf_timestamp": 2488, "host-ts": 2488}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2482 (Wed Jul 25 04:19:30 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2483 (Wed Jul 25 04:19:30 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "470808b8", "local_conf_timestamp": 2483, "host-ts": 2482}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:6fb40f9a-5636-47d5-a5e7-cf1835987996:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:6fb40f9a-5636-47d5-a5e7-cf1835987996:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running f5dc5560 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f5dc5560 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command f5dc5560 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2498 (Wed Jul 25 04:19:46 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2499 (Wed Jul 25 04:19:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "0a380d85", "local_conf_timestamp": 2499, "host-ts": 2498}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2492 (Wed Jul 25 04:19:39 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2492 (Wed Jul 25 04:19:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1e24fbe7", "local_conf_timestamp": 2492, "host-ts": 2492}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:54953ae8-9f55-4cb1-bb4c-7ec68dc9879f:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:54953ae8-9f55-4cb1-bb4c-7ec68dc9879f:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running f99d7d32 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f99d7d32 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command f99d7d32 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2498 (Wed Jul 25 04:19:46 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2499 (Wed Jul 25 04:19:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "0a380d85", "local_conf_timestamp": 2499, "host-ts": 2498}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2492 (Wed Jul 25 04:19:39 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2492 (Wed Jul 25 04:19:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1e24fbe7", "local_conf_timestamp": 2492, "host-ts": 2492}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:2d207ff4-7778-45e7-abe3-56af39bec1c8:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:2d207ff4-7778-45e7-abe3-56af39bec1c8:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running fd26f32a on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fd26f32a on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command fd26f32a on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2508 (Wed Jul 25 04:19:55 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2508 (Wed Jul 25 04:19:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ef34089b", "local_conf_timestamp": 2508, "host-ts": 2508}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2502 (Wed Jul 25 04:19:49 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2502 (Wed Jul 25 04:19:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "78b671f0", "local_conf_timestamp": 2502, "host-ts": 2502}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:4819a00d-a93c-4b9d-9439-77204d723a0a:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:4819a00d-a93c-4b9d-9439-77204d723a0a:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 00ce0cca on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 00ce0cca on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 00ce0cca on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2508 (Wed Jul 25 04:19:55 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2508 (Wed Jul 25 04:19:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ef34089b", "local_conf_timestamp": 2508, "host-ts": 2508}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2502 (Wed Jul 25 04:19:49 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2502 (Wed Jul 25 04:19:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "78b671f0", "local_conf_timestamp": 2502, "host-ts": 2502}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:2eadab93-166a-4267-8a57-0d457b253235:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:2eadab93-166a-4267-8a57-0d457b253235:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 04474b64 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 04474b64 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 04474b64 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2518 (Wed Jul 25 04:20:06 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2519 (Wed Jul 25 04:20:06 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f3fa6dd6", "local_conf_timestamp": 2519, "host-ts": 2518}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2512 (Wed Jul 25 04:19:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2512 (Wed Jul 25 04:19:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "569db239", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:697a24f7-bb86-4e63-abc3-0ee588786d0a:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:697a24f7-bb86-4e63-abc3-0ee588786d0a:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 07152f64 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 07152f64 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 07152f64 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2518 (Wed Jul 25 04:20:06 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2519 (Wed Jul 25 04:20:06 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f3fa6dd6", "local_conf_timestamp": 2519, "host-ts": 2518}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2512 (Wed Jul 25 04:19:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2512 (Wed Jul 25 04:19:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "eng\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f2f900dd140>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/008_restart_he_vm.py", line 165, in <lambda>\n for k, v in _get_he_status(host).items()\n File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/008_restart_he_vm.py", line 106, in _get_he_status\n raise RuntimeError(\'could not parse JSON: %s\' % ret.out)\nRuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2518 (Wed Jul 25 04:20:06 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2519 (Wed Jul 25 04:20:06 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f3fa6dd6", "local_conf_timestamp": 2519, "host-ts": 2518}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2512 (Wed Jul 25 04:19:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2512 (Wed Jul 25 04:19:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "eng\n--------------------- >> end captured logging << ---------------------'

1 1

[JIRA] (OVIRT-2368) Please create some ovirt-ansible-* github repositories
by Barak Korren (oVirt JIRA) 25 Jul '18

25 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2368?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-2368: ------------------------------------- Repos created: * [ovirt-ansible-dump-db|https://github.com/oVirt/ovirt-ansible-dump-db] * [ovirt-ansible-rename-db|https://github.com/oVirt/ovirt-ansible-rename-db] * [ovirt-ansible-remote-db|https://github.com/oVirt/ovirt-ansible-remote-db] * [ovirt-ansible-local-to-remote-db|https://github.com/oVirt/ovirt-ansible-loc…] [~lleistne] please confirm everything works as you requested. > Please create some ovirt-ansible-* github repositories > ------------------------------------------------------ > > Key: OVIRT-2368 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2368 > Project: oVirt - virtualization made easy > Issue Type: Task > Components: GitHub > Reporter: Lucie Leistnerova > Assignee: infra > > Please create this github repositories: > ovirt-ansible-dump-db > ovirt-ansible-rename-db > ovirt-ansible-remote-db > ovirt-ansible-local-to-remote-db > And assign leistnerova as administrator. > Thank you. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2368) Please create some ovirt-ansible-* github repositories
by Barak Korren (oVirt JIRA) 25 Jul '18

25 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2368?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-2368: -------------------------------- Component/s: GitHub > Please create some ovirt-ansible-* github repositories > ------------------------------------------------------ > > Key: OVIRT-2368 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2368 > Project: oVirt - virtualization made easy > Issue Type: Task > Components: GitHub > Reporter: Lucie Leistnerova > Assignee: infra > > Please create this github repositories: > ovirt-ansible-dump-db > ovirt-ansible-rename-db > ovirt-ansible-remote-db > ovirt-ansible-local-to-remote-db > And assign leistnerova as administrator. > Thank you. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2370) Please create ovirt-ansible-shutdown-env github repository
by Barak Korren (oVirt JIRA) 25 Jul '18

25 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2370?page=com.atlassian.jira.… ] Barak Korren reassigned OVIRT-2370: ----------------------------------- Assignee: Barak Korren (was: infra) > Please create ovirt-ansible-shutdown-env github repository > ---------------------------------------------------------- > > Key: OVIRT-2370 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2370 > Project: oVirt - virtualization made easy > Issue Type: Task > Components: GitHub > Reporter: Simone Tiraboschi > Assignee: Barak Korren > > Please create ovirt-ansible-shutdown-env github repository. > Please assign tiraboschi, irosenzw, didib and sandrobonazzola as administrators. > Thanks. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[CQ]: 93154,6 (vdsm) failed "ovirt-master" system tests
by oVirt Jenkins 25 Jul '18

25 Jul '18

Change 93154,6 (vdsm) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/93154/6 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8766/

3 2

s390x (mainframe) CI node is currently not available
by Barak Korren 25 Jul '18

25 Jul '18

Hi all, Unfortunately, the s390x build host we've been using had gone off-line a couple of days ago, and the ETA for having it back up is unknown. The implication of having the host down is that projects that build on s390x have their builds be stuck forever waiting for the s390x node to become available. Sine the s390x builds are missing, builds for other platforms, while being successful do not get passed to the change queue and as a result are not being released to the nightly repos. As a work around for the current situation, we need to disable the s390x builds. Here is a patch that does this for vdsm: https://gerrit.ovirt.org/c/93316/ For V2 projects the change should be done in the projects own repo, for example here is how we did it for the CI jobs of the 'jenkins' repo itself: https://gerrit.ovirt.org/c/93288/ -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

1 0

[JIRA] (OVIRT-2363) 008_basic_ui_sanity.start_grid failure with error "[Errno 2] No such file or directory"
by Barak Korren (oVirt JIRA) 25 Jul '18

25 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2363?page=com.atlassian.jira.… ] Barak Korren edited comment on OVIRT-2363 at 7/25/18 11:20 AM: --------------------------------------------------------------- [~dron] loking at the failed test: {code} [Errno 2] No such file or directory -------------------- >> begin captured stdout << --------------------- executing shell: docker ps --------------------- >> end captured stdout << ---------------------- {code} so inndeed looks like the same issue. I thing that generally 008_basic_ui_sanity.start_grid cannot fail on anything else since all it does is start the Selenium containers. Anyway I think enough time have passed so that all mock caches that have been built with docker missing should have been replaced by now. was (Author: bkorren(a)redhat.com) [~dron] loking at the failed test: {{code}} [Errno 2] No such file or directory -------------------- >> begin captured stdout << --------------------- executing shell: docker ps --------------------- >> end captured stdout << ---------------------- {{code} so inndeed looks like the same issue. I thing that generally 008_basic_ui_sanity.start_grid cannot fail on anything else since all it does is start the Selenium containers. Anyway I think enough time have passed so that all mock caches that have been built with docker missing should have been replaced by now. > 008_basic_ui_sanity.start_grid failure with error "[Errno 2] No such file or directory" > --------------------------------------------------------------------------------------- > > Key: OVIRT-2363 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2363 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_failures > > patch that failed is: https://gerrit.ovirt.org/#/c/92882/ - core: hotplug new disk only if VM is up or paused states > I don't think the patch causes a regression but I am looking at the logs and checking to see if there was a behavior change which caused the ost test itself to fail. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2363) 008_basic_ui_sanity.start_grid failure with error "[Errno 2] No such file or directory"
by Barak Korren (oVirt JIRA) 25 Jul '18

25 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2363?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-2363: ------------------------------------- [~dron] loking at the failed test: {{code}} [Errno 2] No such file or directory -------------------- >> begin captured stdout << --------------------- executing shell: docker ps --------------------- >> end captured stdout << ---------------------- {{code} so inndeed looks like the same issue. I thing that generally 008_basic_ui_sanity.start_grid cannot fail on anything else since all it does is start the Selenium containers. Anyway I think enough time have passed so that all mock caches that have been built with docker missing should have been replaced by now. > 008_basic_ui_sanity.start_grid failure with error "[Errno 2] No such file or directory" > --------------------------------------------------------------------------------------- > > Key: OVIRT-2363 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2363 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_failures > > patch that failed is: https://gerrit.ovirt.org/#/c/92882/ - core: hotplug new disk only if VM is up or paused states > I don't think the patch causes a regression but I am looking at the logs and checking to see if there was a behavior change which caused the ost test itself to fail. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2363) 008_basic_ui_sanity.start_grid failure with error "[Errno 2] No such file or directory"
by Dafna Ron (oVirt JIRA) 25 Jul '18

25 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2363?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2363: ---------------------------------- looking at the list [~bkorren(a)redhat.com] wrote this is related to docker not being installed. Barak , can you confirm this is the case as well? > 008_basic_ui_sanity.start_grid failure with error "[Errno 2] No such file or directory" > --------------------------------------------------------------------------------------- > > Key: OVIRT-2363 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2363 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_failures > > patch that failed is: https://gerrit.ovirt.org/#/c/92882/ - core: hotplug new disk only if VM is up or paused states > I don't think the patch causes a regression but I am looking at the logs and checking to see if there was a behavior change which caused the ost test itself to fail. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[oVirt Jenkins] ovirt-node-ng_master_build-artifacts-el7-x86_64 - Build # 867 - Failure!
by jenkins＠jenkins.phx.ovirt.org 25 Jul '18

25 Jul '18

Project: http://jenkins.ovirt.org/job/ovirt-node-ng_master_build-artifacts-el7-x86_6… Build: http://jenkins.ovirt.org/job/ovirt-node-ng_master_build-artifacts-el7-x86_6… Build Number: 867 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #867 [Sandro Bonazzola] add kmajumde(a)redhat.com [Barak Korren] Refactor: consolidate STDCI reporting code [Barak Korren] Enable customizing GitHub report URL from DSL [Daniel Belenky] Temporary disable s390x jobs [Yuval Turgeman] nodectl: package for python3 ----------------- Failed Tests: ----------------- No tests ran.

1 2

[CQ]: 93081,3 (ovirt-imageio) failed "ovirt-master" system tests
by oVirt Jenkins 25 Jul '18

25 Jul '18

Change 93081,3 (ovirt-imageio) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/93081/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8778/

2 1

[JIRA] (OVIRT-2363) 008_basic_ui_sanity.start_grid failure with error "[Errno 2] No such file or directory"
by Dafna Ron (oVirt JIRA) 25 Jul '18

25 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2363?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2363: ---------------------------------- we have a second failure on this test. this time for project ovirt-imageio. looking at the logs they seem identical to the original issue Here is the failure: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8778/ I moved priority of this issue to highest and adding [~dfediuck] and [~gbenhaim(a)redhat.com] since I would like a second opinion from Gal on weather he thinks its a regression or if this can be ost related or a regression. > 008_basic_ui_sanity.start_grid failure with error "[Errno 2] No such file or directory" > --------------------------------------------------------------------------------------- > > Key: OVIRT-2363 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2363 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_failures > > patch that failed is: https://gerrit.ovirt.org/#/c/92882/ - core: hotplug new disk only if VM is up or paused states > I don't think the patch causes a regression but I am looking at the logs and checking to see if there was a behavior change which caused the ost test itself to fail. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2363) 008_basic_ui_sanity.start_grid failure with error "[Errno 2] No such file or directory"
by Eyal Edri (oVirt JIRA) 25 Jul '18

25 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2363?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2363: ---------------------------------- [~dron] is this currently failing master constantly? ( also, please include the oVirt version in the ticket summary ) > 008_basic_ui_sanity.start_grid failure with error "[Errno 2] No such file or directory" > --------------------------------------------------------------------------------------- > > Key: OVIRT-2363 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2363 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_failures > > patch that failed is: https://gerrit.ovirt.org/#/c/92882/ - core: hotplug new disk only if VM is up or paused states > I don't think the patch causes a regression but I am looking at the logs and checking to see if there was a behavior change which caused the ost test itself to fail. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2363) 008_basic_ui_sanity.start_grid failure with error "[Errno 2] No such file or directory"
by Dafna Ron (oVirt JIRA) 25 Jul '18

25 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2363?page=com.atlassian.jira.… ] Dafna Ron updated OVIRT-2363: ----------------------------- Priority: Highest (was: Medium) > 008_basic_ui_sanity.start_grid failure with error "[Errno 2] No such file or directory" > --------------------------------------------------------------------------------------- > > Key: OVIRT-2363 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2363 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_failures > > patch that failed is: https://gerrit.ovirt.org/#/c/92882/ - core: hotplug new disk only if VM is up or paused states > I don't think the patch causes a regression but I am looking at the logs and checking to see if there was a behavior change which caused the ost test itself to fail. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[CQ]: 93155, 7 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 25 Jul '18

25 Jul '18

A system test invoked by the "ovirt-master" change queue including change 93155,7 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 93154,6 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 93154,6 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/93155/7 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/93154/6 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8773/

2 1

[CQ]: 93274,2 (ioprocess) failed "ovirt-4.2" system tests
by oVirt Jenkins 25 Jul '18

25 Jul '18

Change 93274,2 (ioprocess) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/93274/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2706/

2 1

[JIRA] (OVIRT-2370) Please create ovirt-ansible-shutdown-env github repository
by Simone Tiraboschi (oVirt JIRA) 25 Jul '18

25 Jul '18

Simone Tiraboschi created OVIRT-2370: ---------------------------------------- Summary: Please create ovirt-ansible-shutdown-env github repository Key: OVIRT-2370 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2370 Project: oVirt - virtualization made easy Issue Type: Task Components: GitHub Reporter: Simone Tiraboschi Assignee: infra Please create ovirt-ansible-shutdown-env github repository. Please assign tiraboschi, irosenzw, didib and sandrobonazzola as administrators. Thanks. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2316) Docker containers disk space is limited to 10GB by default
by Daniel Belenky (oVirt JIRA) 25 Jul '18

25 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2316?page=com.atlassian.jira.… ] Daniel Belenky commented on OVIRT-2316: --------------------------------------- We need to decide on a formula to calculate how much disk space to allocate for each container. Probably something like {code:java} ($TOTAL-$HOST_RESERVED)/$NUM_OF_CONTAINERS {code} > Docker containers disk space is limited to 10GB by default > ---------------------------------------------------------- > > Key: OVIRT-2316 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2316 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Gal Ben Haim > Assignee: infra > -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2369) Make sure that $WORKSPACE is the same everywhere
by Daniel Belenky (oVirt JIRA) 25 Jul '18

25 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2369?page=com.atlassian.jira.… ] Daniel Belenky updated OVIRT-2369: ---------------------------------- Assignee: Daniel Belenky (was: infra) Status: In Progress (was: To Do) > Make sure that $WORKSPACE is the same everywhere > ------------------------------------------------ > > Key: OVIRT-2369 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2369 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Daniel Belenky > Assignee: Daniel Belenky > > We need to make sure that the $WORKSPACE leads to the same path in the host, and in mock users to mount volumes to their containers. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2328) Mount cache dirs in stdci container
by Daniel Belenky (oVirt JIRA) 25 Jul '18

25 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2328?page=com.atlassian.jira.… ] Daniel Belenky updated OVIRT-2328: ---------------------------------- Assignee: Daniel Belenky (was: infra) Status: In Progress (was: To Do) > Mount cache dirs in stdci container > ----------------------------------- > > Key: OVIRT-2328 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2328 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Daniel Belenky > Assignee: Daniel Belenky > Labels: POC > > In order to utilize current caching mechanisms, we need to mount /var/lib/ago and /var/cache/mock to stdci container -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2369) Make sure that $WORKSPACE is the same everywhere
by Daniel Belenky (oVirt JIRA) 25 Jul '18

25 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2369?page=com.atlassian.jira.… ] Daniel Belenky updated OVIRT-2369: ---------------------------------- Description: We need to make sure that the $WORKSPACE leads to the same path in the host, and in mock users to mount volumes to their containers. > Make sure that $WORKSPACE is the same everywhere > ------------------------------------------------ > > Key: OVIRT-2369 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2369 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Daniel Belenky > Assignee: infra > > We need to make sure that the $WORKSPACE leads to the same path in the host, and in mock users to mount volumes to their containers. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2369) Make sure that $WORKSPACE is the same everywhere
by Daniel Belenky (oVirt JIRA) 25 Jul '18

25 Jul '18

Daniel Belenky created OVIRT-2369: ------------------------------------- Summary: Make sure that $WORKSPACE is the same everywhere Key: OVIRT-2369 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2369 Project: oVirt - virtualization made easy Issue Type: Task Reporter: Daniel Belenky Assignee: infra -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0