July 2018 - Infra - oVirt List Archives

[JIRA] (OVIRT-2363) 008_basic_ui_sanity.start_grid failure with error "[Errno 2] No such file or directory"
by Barak Korren (oVirt JIRA) 25 Jul '18

25 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2363?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-2363: ------------------------------------- no, probably just need to erase mock cache on the hosts where this is happening to get a new one created with docker installed. > 008_basic_ui_sanity.start_grid failure with error "[Errno 2] No such file or directory" > --------------------------------------------------------------------------------------- > > Key: OVIRT-2363 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2363 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_failures > > patch that failed is: https://gerrit.ovirt.org/#/c/92882/ - core: hotplug new disk only if VM is up or paused states > I don't think the patch causes a regression but I am looking at the logs and checking to see if there was a behavior change which caused the ost test itself to fail. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[oVirt Jenkins] ovirt-system-tests_compat-4.1-suite-master - Build # 2 - Failure!
by jenkins＠jenkins.phx.ovirt.org 25 Jul '18

25 Jul '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.1-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.1-suite-master/2/ Build Number: 2 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #2 [Lev Veyde] 004_basic_sanity.py cleanup ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.preview_snapshot_with_memory_41 Error Message: False != True after 600 seconds Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 78, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/workspace/ovirt-system-tests_compat-4.1-suite-master/ovirt-system-tests/compat-4.1-suite-master/test-scenarios/004_basic_sanity.py", line 431, in preview_snapshot_with_memory pass File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ self.gen.next() File "/home/jenkins/workspace/ovirt-system-tests_compat-4.1-suite-master/ovirt-system-tests/compat-4.1-suite-master/test_utils/__init__.py", line 264, in TestEvent lambda: File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 271, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 258, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 237, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 600 seconds

1 8

[CQ]: 89641,14 (ovirt-engine) failed "ovirt-master" system tests
by oVirt Jenkins 25 Jul '18

25 Jul '18

Change 89641,14 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/89641/14 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8790/

1 0

oVirt infra weekly meeting notes 25.07.2018
by Evgheni Dereveanchin 25 Jul '18

25 Jul '18

Hi everyone, Please find the topics of this week's infra meeting below: *PHX: - Jenkins updated OVIRT-2262 <https://ovirt-jira.atlassian.net/browse/OVIRT-2262>- Let’s Encrypt SSL certificate installed on PHX engine OVIRT-531 <https://ovirt-jira.atlassian.net/browse/OVIRT-531>- 3 hosts rebuilt as bare metals OVIRT-2306 <https://ovirt-jira.atlassian.net/browse/OVIRT-2306>- 2 hosts upgraded to CentOS 7.5, rest on the way- Resources server crash reported OVIRT-2338 <https://ovirt-jira.atlassian.net/browse/OVIRT-2338>- Network flop between RDU and PHX caused monitoring alerts OVIRT-2356 <https://ovirt-jira.atlassian.net/browse/OVIRT-2356>- Patch sent to update PHX oVirt setup RTD page https://gerrit.ovirt.org/#/c/93166/ <https://gerrit.ovirt.org/#/c/93166/> Other topics: - Website moved to OSAS OVIRT-1799 <https://ovirt-jira.atlassian.net/browse/OVIRT-1799>- s390x slave down - builds for this arch temporarily disabledAI: - Shut down old website VM (ensure we’re monitoring the right one)- Document Let’s encrypt on PHX engine for DR purposes- Provide IRC nicks for direct contacts upstream* -- Regards, Evgheni Dereveanchin

1 0

[CQ]: 92924, 14 (ovirt-provider-ovn) failed "ovirt-master" system tests
by oVirt Jenkins 25 Jul '18

25 Jul '18

Change 92924,14 (ovirt-provider-ovn) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/92924/14 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8782/

2 1

[JIRA] (OVIRT-2363) 008_basic_ui_sanity.start_grid failure with error "[Errno 2] No such file or directory"
by Dafna Ron (oVirt JIRA) 25 Jul '18

25 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2363?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2363: ---------------------------------- third project to fail on this: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8782/ [~bkorren(a)redhat.com] [~ederevea] any chance that we have a new issue that is causing the same failures? > 008_basic_ui_sanity.start_grid failure with error "[Errno 2] No such file or directory" > --------------------------------------------------------------------------------------- > > Key: OVIRT-2363 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2363 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_failures > > patch that failed is: https://gerrit.ovirt.org/#/c/92882/ - core: hotplug new disk only if VM is up or paused states > I don't think the patch causes a regression but I am looking at the logs and checking to see if there was a behavior change which caused the ost test itself to fail. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-master - Build # 222 - Failure!
by jenkins＠jenkins.phx.ovirt.org 25 Jul '18

25 Jul '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/222/ Build Number: 222 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #222 [Milan Zamazal] Wait for snapshot events correctly in preview_snapshot_with_memory ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 008_restart_he_vm.restart_he_vm Error Message: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2518 (Wed Jul 25 04:20:06 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2519 (Wed Jul 25 04:20:06 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f3fa6dd6", "local_conf_timestamp": 2519, "host-ts": 2518}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2512 (Wed Jul 25 04:19:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2512 (Wed Jul 25 04:19:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "eng -------------------- >> begin captured logging << -------------------- lago.ssh: DEBUG: start task:932dbfdb-8f78-42c6-9b1e-6aadee5d8091:Get ssh client for lago-he-node-ng-suite-master-host-1: lago.ssh: DEBUG: end task:932dbfdb-8f78-42c6-9b1e-6aadee5d8091:Get ssh client for lago-he-node-ng-suite-master-host-1: lago.ssh: DEBUG: Running 7763a99a on lago-he-node-ng-suite-master-host-1: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 7763a99a on lago-he-node-ng-suite-master-host-1 returned with 0 lago.ssh: DEBUG: Command 7763a99a on lago-he-node-ng-suite-master-host-1 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2275 (Wed Jul 25 04:16:03 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2275 (Wed Jul 25 04:16:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "634493fe", "local_conf_timestamp": 2275, "host-ts": 2275}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2281 (Wed Jul 25 04:16:08 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2281 (Wed Jul 25 04:16:08 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69ca08d9", "local_conf_timestamp": 2281, "host-ts": 2281}, "global_maintenance": true} root: INFO: * Shutting down HE VM on host: lago-he-node-ng-suite-master-host-0 lago.ssh: DEBUG: start task:2bd52306-1957-43f5-a4cd-a16a4223bbd2:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:2bd52306-1957-43f5-a4cd-a16a4223bbd2:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 78525040 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-shutdown lago.ssh: DEBUG: Command 78525040 on lago-he-node-ng-suite-master-host-0 returned with 0 root: INFO: * Command succeeded root: INFO: * Waiting for VM to be down... lago.ssh: DEBUG: start task:75e52da9-c741-4089-8bda-708623dbb4cc:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:75e52da9-c741-4089-8bda-708623dbb4cc:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 797baba6 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 797baba6 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 797baba6 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2285 (Wed Jul 25 04:16:13 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2286 (Wed Jul 25 04:16:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "bc8dc98f", "local_conf_timestamp": 2286, "host-ts": 2285}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2281 (Wed Jul 25 04:16:08 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2281 (Wed Jul 25 04:16:08 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69ca08d9", "local_conf_timestamp": 2281, "host-ts": 2281}, "global_maintenance": true} lago.ssh: DEBUG: start task:ac00a93a-e808-4f72-bb6f-f29c7bc259a6:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:ac00a93a-e808-4f72-bb6f-f29c7bc259a6:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 812bb044 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 812bb044 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 812bb044 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2295 (Wed Jul 25 04:16:22 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2295 (Wed Jul 25 04:16:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf71ee59", "local_conf_timestamp": 2295, "host-ts": 2295}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2291 (Wed Jul 25 04:16:18 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2291 (Wed Jul 25 04:16:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "47e1cb10", "local_conf_timestamp": 2291, "host-ts": 2291}, "global_maintenance": true} lago.ssh: DEBUG: start task:bb5c3ebf-f315-4147-9664-83fbdfb0b5e8:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:bb5c3ebf-f315-4147-9664-83fbdfb0b5e8:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 83e14d9e on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 83e14d9e on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 83e14d9e on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2305 (Wed Jul 25 04:16:33 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2305 (Wed Jul 25 04:16:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "16c42d10", "local_conf_timestamp": 2305, "host-ts": 2305}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2301 (Wed Jul 25 04:16:28 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2301 (Wed Jul 25 04:16:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "21734107", "local_conf_timestamp": 2301, "host-ts": 2301}, "global_maintenance": true} root: INFO: * VM is down. root: INFO: * Stopping services... lago.ssh: DEBUG: start task:a91ce735-904b-4589-8d49-374d4d4e5233:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:a91ce735-904b-4589-8d49-374d4d4e5233:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 84cdbff8 on lago-he-node-ng-suite-master-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command 84cdbff8 on lago-he-node-ng-suite-master-host-0 returned with 0 root: INFO: * Starting services... lago.ssh: DEBUG: start task:69e2b63b-31fc-464e-b7ff-c499a95be86e:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:69e2b63b-31fc-464e-b7ff-c499a95be86e:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 8dac9a18 on lago-he-node-ng-suite-master-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command 8dac9a18 on lago-he-node-ng-suite-master-host-0 returned with 0 root: INFO: * Waiting for agent to be ready... lago.ssh: DEBUG: start task:5c41febb-2a25-4889-a7c1-5c9da7a33f96:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:5c41febb-2a25-4889-a7c1-5c9da7a33f96:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 909bffb6 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 909bffb6 on lago-he-node-ng-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 909bffb6 on lago-he-node-ng-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:56d0e0c8-3a30-4b6e-acf5-ba1e387ceb7c:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:56d0e0c8-3a30-4b6e-acf5-ba1e387ceb7c:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 977e0a9a on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 977e0a9a on lago-he-node-ng-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 977e0a9a on lago-he-node-ng-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:4b36c31c-c6b7-4bbb-b038-b67e2a26e788:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:4b36c31c-c6b7-4bbb-b038-b67e2a26e788:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 9a1513fc on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 9a1513fc on lago-he-node-ng-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 9a1513fc on lago-he-node-ng-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:e3e68036-c86b-4a3e-ad6d-788272025c42:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:e3e68036-c86b-4a3e-ad6d-788272025c42:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 9cbc9a94 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 9cbc9a94 on lago-he-node-ng-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 9cbc9a94 on lago-he-node-ng-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:7b9f8a19-b7dc-4d00-9202-e07d88888dfe:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:7b9f8a19-b7dc-4d00-9202-e07d88888dfe:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 9f7865c4 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 9f7865c4 on lago-he-node-ng-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 9f7865c4 on lago-he-node-ng-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:1944f900-1947-46ca-b701-3f71d3025a23:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:1944f900-1947-46ca-b701-3f71d3025a23:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running a212e07a on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command a212e07a on lago-he-node-ng-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command a212e07a on lago-he-node-ng-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:289afe1e-8a48-4e1f-b585-c7afe283cc2a:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:289afe1e-8a48-4e1f-b585-c7afe283cc2a:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running a4b92d66 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command a4b92d66 on lago-he-node-ng-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command a4b92d66 on lago-he-node-ng-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:65b385d4-2abe-4c8a-9f9f-37904c3320b5:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:65b385d4-2abe-4c8a-9f9f-37904c3320b5:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running a755a3f6 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command a755a3f6 on lago-he-node-ng-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command a755a3f6 on lago-he-node-ng-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:179389d5-7a94-4742-9328-73c41ed07ed0:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:179389d5-7a94-4742-9328-73c41ed07ed0:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running a9fe6f48 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command a9fe6f48 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command a9fe6f48 on lago-he-node-ng-suite-master-host-0 output: !! Cluster is in GLOBAL MAINTENANCE mode !! --== Host lago-he-node-ng-suite-master-host-0 (id: 1) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : lago-he-node-ng-suite-master-host-0 Host ID : 1 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"} Score : 0 stopped : False Local maintenance : False crc32 : 744e472a local_conf_timestamp : 2368 Host timestamp : 2367 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2367 (Wed Jul 25 04:17:35 2018) host-id=1 score=0 vm_conf_refresh_time=2368 (Wed Jul 25 04:17:35 2018) conf_on_shared_storage=True maintenance=False state=ReinitializeFSM stopped=False --== Host lago-he-node-ng-suite-master-host-1 (id: 2) status ==-- conf_on_shared_storage : True Status up-to-date : False Hostname : lago-he-node-ng-suite-master-host-1 Host ID : 2 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : c58bc9b1 local_conf_timestamp : 2361 Host timestamp : 2361 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2361 (Wed Jul 25 04:17:28 2018) host-id=2 score=3400 vm_conf_refresh_time=2361 (Wed Jul 25 04:17:29 2018) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False !! Cluster is in GLOBAL MAINTENANCE mode !! root: INFO: * Agent is ready. root: INFO: * Starting VM... lago.ssh: DEBUG: start task:771d13e3-aa83-4b0c-ad2f-73254bee44b5:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:771d13e3-aa83-4b0c-ad2f-73254bee44b5:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running ab02495a on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-start lago.ssh: DEBUG: Command ab02495a on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command ab02495a on lago-he-node-ng-suite-master-host-0 output: VM exists and is down, cleaning up and restarting root: INFO: * Command succeeded root: INFO: * Waiting for VM to be UP... lago.ssh: DEBUG: start task:9b23a87d-00d7-4b1b-9df3-d4f2a93a347a:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:9b23a87d-00d7-4b1b-9df3-d4f2a93a347a:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running ad1c960a on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ad1c960a on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command ad1c960a on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2367 (Wed Jul 25 04:17:35 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=2368 (Wed Jul 25 04:17:35 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=ReinitializeFSM\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "744e472a", "local_conf_timestamp": 2368, "host-ts": 2367}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2361 (Wed Jul 25 04:17:28 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2361 (Wed Jul 25 04:17:29 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c58bc9b1", "local_conf_timestamp": 2361, "host-ts": 2361}, "global_maintenance": true} lago.ssh: DEBUG: start task:38723e85-fc65-4a4c-903e-49754f691574:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:38723e85-fc65-4a4c-903e-49754f691574:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running b40283bc on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b40283bc on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command b40283bc on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2377 (Wed Jul 25 04:17:44 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2377 (Wed Jul 25 04:17:45 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "40391805", "local_conf_timestamp": 2377, "host-ts": 2377}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2381 (Wed Jul 25 04:17:48 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2381 (Wed Jul 25 04:17:49 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8b5c590e", "local_conf_timestamp": 2381, "host-ts": 2381}, "global_maintenance": true} lago.ssh: DEBUG: start task:cd1092f6-e8c6-40ee-99d2-6a6edd7be1b5:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:cd1092f6-e8c6-40ee-99d2-6a6edd7be1b5:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running b6c56286 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b6c56286 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command b6c56286 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2387 (Wed Jul 25 04:17:55 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2388 (Wed Jul 25 04:17:55 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d2ce9013", "local_conf_timestamp": 2388, "host-ts": 2387}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2381 (Wed Jul 25 04:17:48 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2381 (Wed Jul 25 04:17:49 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8b5c590e", "local_conf_timestamp": 2381, "host-ts": 2381}, "global_maintenance": true} root: INFO: * VM is UP. root: INFO: * Waiting for engine to start... lago.ssh: DEBUG: start task:567f9428-43b1-44f5-ad18-0f4a01c91b13:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:567f9428-43b1-44f5-ad18-0f4a01c91b13:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running b7c81b4c on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b7c81b4c on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command b7c81b4c on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2387 (Wed Jul 25 04:17:55 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2388 (Wed Jul 25 04:17:55 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d2ce9013", "local_conf_timestamp": 2388, "host-ts": 2387}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2381 (Wed Jul 25 04:17:48 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2381 (Wed Jul 25 04:17:49 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8b5c590e", "local_conf_timestamp": 2381, "host-ts": 2381}, "global_maintenance": true} lago.ssh: DEBUG: start task:029af7ab-0480-465a-ad7d-9d3a37b5b355:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:029af7ab-0480-465a-ad7d-9d3a37b5b355:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running bec6f814 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command bec6f814 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command bec6f814 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2398 (Wed Jul 25 04:18:05 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2398 (Wed Jul 25 04:18:06 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "412e4ba9", "local_conf_timestamp": 2398, "host-ts": 2398}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2391 (Wed Jul 25 04:17:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2392 (Wed Jul 25 04:17:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "3c7ecb7d", "local_conf_timestamp": 2392, "host-ts": 2391}, "global_maintenance": true} lago.ssh: DEBUG: start task:13363bf6-bedc-4e09-814b-bf03d9839892:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:13363bf6-bedc-4e09-814b-bf03d9839892:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running c1a71ce4 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command c1a71ce4 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command c1a71ce4 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2408 (Wed Jul 25 04:18:16 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2408 (Wed Jul 25 04:18:16 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9432151", "local_conf_timestamp": 2408, "host-ts": 2408}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2402 (Wed Jul 25 04:18:09 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2402 (Wed Jul 25 04:18:09 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9a202027", "local_conf_timestamp": 2402, "host-ts": 2402}, "global_maintenance": true} lago.ssh: DEBUG: start task:8e86ea8a-ed97-49c2-aa31-0d07662c480f:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:8e86ea8a-ed97-49c2-aa31-0d07662c480f:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running c4dffe62 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command c4dffe62 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command c4dffe62 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2408 (Wed Jul 25 04:18:16 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2408 (Wed Jul 25 04:18:16 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9432151", "local_conf_timestamp": 2408, "host-ts": 2408}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2402 (Wed Jul 25 04:18:09 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2402 (Wed Jul 25 04:18:09 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9a202027", "local_conf_timestamp": 2402, "host-ts": 2402}, "global_maintenance": true} lago.ssh: DEBUG: start task:e801a931-3f5c-463c-b00e-07905169e39b:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:e801a931-3f5c-463c-b00e-07905169e39b:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running c86e9048 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command c86e9048 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command c86e9048 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2418 (Wed Jul 25 04:18:26 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2419 (Wed Jul 25 04:18:26 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "625c3f40", "local_conf_timestamp": 2419, "host-ts": 2418}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2412 (Wed Jul 25 04:18:20 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2413 (Wed Jul 25 04:18:20 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c30cd378", "local_conf_timestamp": 2413, "host-ts": 2412}, "global_maintenance": true} lago.ssh: DEBUG: start task:fc9d9c05-cc70-49f4-9195-5801981130da:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:fc9d9c05-cc70-49f4-9195-5801981130da:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running cb60cc6c on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command cb60cc6c on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command cb60cc6c on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2418 (Wed Jul 25 04:18:26 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2419 (Wed Jul 25 04:18:26 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "625c3f40", "local_conf_timestamp": 2419, "host-ts": 2418}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2412 (Wed Jul 25 04:18:20 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2413 (Wed Jul 25 04:18:20 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c30cd378", "local_conf_timestamp": 2413, "host-ts": 2412}, "global_maintenance": true} lago.ssh: DEBUG: start task:a08eb28b-2662-4ab3-a80b-77ef2581a073:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:a08eb28b-2662-4ab3-a80b-77ef2581a073:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running cee93edc on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command cee93edc on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command cee93edc on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2428 (Wed Jul 25 04:18:35 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2428 (Wed Jul 25 04:18:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ace23939", "local_conf_timestamp": 2428, "host-ts": 2428}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2422 (Wed Jul 25 04:18:29 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2422 (Wed Jul 25 04:18:29 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c677a7b5", "local_conf_timestamp": 2422, "host-ts": 2422}, "global_maintenance": true} lago.ssh: DEBUG: start task:5475c36c-7873-4bb4-b0a4-89b7a176ed10:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:5475c36c-7873-4bb4-b0a4-89b7a176ed10:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running d2517044 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command d2517044 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command d2517044 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2438 (Wed Jul 25 04:18:46 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2439 (Wed Jul 25 04:18:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b02c5c74", "local_conf_timestamp": 2439, "host-ts": 2438}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2432 (Wed Jul 25 04:18:39 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2432 (Wed Jul 25 04:18:40 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e85c647c", "local_conf_timestamp": 2432, "host-ts": 2432}, "global_maintenance": true} lago.ssh: DEBUG: start task:378598d4-4368-44ff-b912-b640c5a049b0:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:378598d4-4368-44ff-b912-b640c5a049b0:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running d56c33a4 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command d56c33a4 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command d56c33a4 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2438 (Wed Jul 25 04:18:46 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2439 (Wed Jul 25 04:18:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b02c5c74", "local_conf_timestamp": 2439, "host-ts": 2438}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2432 (Wed Jul 25 04:18:39 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2432 (Wed Jul 25 04:18:40 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e85c647c", "local_conf_timestamp": 2432, "host-ts": 2432}, "global_maintenance": true} lago.ssh: DEBUG: start task:7f5db09d-6b15-4582-8818-3c99d0291ce5:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:7f5db09d-6b15-4582-8818-3c99d0291ce5:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running d863856c on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command d863856c on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command d863856c on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2448 (Wed Jul 25 04:18:55 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2448 (Wed Jul 25 04:18:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "0f24c3a2", "local_conf_timestamp": 2448, "host-ts": 2448}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2442 (Wed Jul 25 04:18:49 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2442 (Wed Jul 25 04:18:49 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "228f2f03", "local_conf_timestamp": 2442, "host-ts": 2442}, "global_maintenance": true} lago.ssh: DEBUG: start task:639271a3-0803-4020-966c-200e8adb5117:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:639271a3-0803-4020-966c-200e8adb5117:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running db6b7cf6 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command db6b7cf6 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command db6b7cf6 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2448 (Wed Jul 25 04:18:55 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2448 (Wed Jul 25 04:18:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "0f24c3a2", "local_conf_timestamp": 2448, "host-ts": 2448}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2442 (Wed Jul 25 04:18:49 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2442 (Wed Jul 25 04:18:49 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "228f2f03", "local_conf_timestamp": 2442, "host-ts": 2442}, "global_maintenance": true} lago.ssh: DEBUG: start task:d1c10391-bcf5-4501-ae6f-47a95d795283:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:d1c10391-bcf5-4501-ae6f-47a95d795283:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running de65a36e on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command de65a36e on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command de65a36e on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2458 (Wed Jul 25 04:19:06 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2458 (Wed Jul 25 04:19:06 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "64ed9679", "local_conf_timestamp": 2458, "host-ts": 2458}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2452 (Wed Jul 25 04:18:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2452 (Wed Jul 25 04:18:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "0ca4ecca", "local_conf_timestamp": 2452, "host-ts": 2452}, "global_maintenance": true} lago.ssh: DEBUG: start task:4e03db00-d1d4-43b0-bcfd-a21d3c32b7a7:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:4e03db00-d1d4-43b0-bcfd-a21d3c32b7a7:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running e173ed40 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e173ed40 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command e173ed40 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2458 (Wed Jul 25 04:19:06 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2458 (Wed Jul 25 04:19:06 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "64ed9679", "local_conf_timestamp": 2458, "host-ts": 2458}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2452 (Wed Jul 25 04:18:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2452 (Wed Jul 25 04:18:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "0ca4ecca", "local_conf_timestamp": 2452, "host-ts": 2452}, "global_maintenance": true} lago.ssh: DEBUG: start task:d8173e8d-c1e7-49d4-8863-b1007ff2704f:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:d8173e8d-c1e7-49d4-8863-b1007ff2704f:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running e462c878 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e462c878 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command e462c878 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2468 (Wed Jul 25 04:19:15 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2468 (Wed Jul 25 04:19:15 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d8b66814", "local_conf_timestamp": 2468, "host-ts": 2468}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2462 (Wed Jul 25 04:19:10 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2463 (Wed Jul 25 04:19:10 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "09df9807", "local_conf_timestamp": 2463, "host-ts": 2462}, "global_maintenance": true} lago.ssh: DEBUG: start task:27e25ae4-3414-4640-8869-9cffad8f6814:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:27e25ae4-3414-4640-8869-9cffad8f6814:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running e794bfec on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e794bfec on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command e794bfec on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2468 (Wed Jul 25 04:19:15 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2468 (Wed Jul 25 04:19:15 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d8b66814", "local_conf_timestamp": 2468, "host-ts": 2468}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2462 (Wed Jul 25 04:19:10 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2463 (Wed Jul 25 04:19:10 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "09df9807", "local_conf_timestamp": 2463, "host-ts": 2462}, "global_maintenance": true} lago.ssh: DEBUG: start task:d23aa18c-4462-4239-8db5-c29feecac71e:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:d23aa18c-4462-4239-8db5-c29feecac71e:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running eb998bd6 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command eb998bd6 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command eb998bd6 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2478 (Wed Jul 25 04:19:26 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2479 (Wed Jul 25 04:19:26 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "41565544", "local_conf_timestamp": 2479, "host-ts": 2478}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2472 (Wed Jul 25 04:19:19 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2472 (Wed Jul 25 04:19:20 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "50f36b58", "local_conf_timestamp": 2472, "host-ts": 2472}, "global_maintenance": true} lago.ssh: DEBUG: start task:37e54f8a-9fe3-47cd-8896-bb560cc5e4d1:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:37e54f8a-9fe3-47cd-8896-bb560cc5e4d1:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running eed21502 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command eed21502 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command eed21502 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2478 (Wed Jul 25 04:19:26 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2479 (Wed Jul 25 04:19:26 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "41565544", "local_conf_timestamp": 2479, "host-ts": 2478}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2472 (Wed Jul 25 04:19:19 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2472 (Wed Jul 25 04:19:20 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "50f36b58", "local_conf_timestamp": 2472, "host-ts": 2472}, "global_maintenance": true} lago.ssh: DEBUG: start task:b6793d51-1124-40aa-9f42-a2107d7649cc:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:b6793d51-1124-40aa-9f42-a2107d7649cc:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running f25fec4e on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f25fec4e on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command f25fec4e on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2488 (Wed Jul 25 04:19:35 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2488 (Wed Jul 25 04:19:35 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "16f668c8", "local_conf_timestamp": 2488, "host-ts": 2488}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2482 (Wed Jul 25 04:19:30 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2483 (Wed Jul 25 04:19:30 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "470808b8", "local_conf_timestamp": 2483, "host-ts": 2482}, "global_maintenance": true} lago.ssh: DEBUG: start task:6fb40f9a-5636-47d5-a5e7-cf1835987996:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:6fb40f9a-5636-47d5-a5e7-cf1835987996:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running f5dc5560 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f5dc5560 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command f5dc5560 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2498 (Wed Jul 25 04:19:46 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2499 (Wed Jul 25 04:19:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "0a380d85", "local_conf_timestamp": 2499, "host-ts": 2498}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2492 (Wed Jul 25 04:19:39 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2492 (Wed Jul 25 04:19:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1e24fbe7", "local_conf_timestamp": 2492, "host-ts": 2492}, "global_maintenance": true} lago.ssh: DEBUG: start task:54953ae8-9f55-4cb1-bb4c-7ec68dc9879f:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:54953ae8-9f55-4cb1-bb4c-7ec68dc9879f:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running f99d7d32 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f99d7d32 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command f99d7d32 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2498 (Wed Jul 25 04:19:46 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2499 (Wed Jul 25 04:19:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "0a380d85", "local_conf_timestamp": 2499, "host-ts": 2498}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2492 (Wed Jul 25 04:19:39 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2492 (Wed Jul 25 04:19:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1e24fbe7", "local_conf_timestamp": 2492, "host-ts": 2492}, "global_maintenance": true} lago.ssh: DEBUG: start task:2d207ff4-7778-45e7-abe3-56af39bec1c8:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:2d207ff4-7778-45e7-abe3-56af39bec1c8:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running fd26f32a on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command fd26f32a on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command fd26f32a on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2508 (Wed Jul 25 04:19:55 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2508 (Wed Jul 25 04:19:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ef34089b", "local_conf_timestamp": 2508, "host-ts": 2508}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2502 (Wed Jul 25 04:19:49 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2502 (Wed Jul 25 04:19:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "78b671f0", "local_conf_timestamp": 2502, "host-ts": 2502}, "global_maintenance": true} lago.ssh: DEBUG: start task:4819a00d-a93c-4b9d-9439-77204d723a0a:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:4819a00d-a93c-4b9d-9439-77204d723a0a:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 00ce0cca on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 00ce0cca on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 00ce0cca on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2508 (Wed Jul 25 04:19:55 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2508 (Wed Jul 25 04:19:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ef34089b", "local_conf_timestamp": 2508, "host-ts": 2508}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2502 (Wed Jul 25 04:19:49 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2502 (Wed Jul 25 04:19:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "78b671f0", "local_conf_timestamp": 2502, "host-ts": 2502}, "global_maintenance": true} lago.ssh: DEBUG: start task:2eadab93-166a-4267-8a57-0d457b253235:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:2eadab93-166a-4267-8a57-0d457b253235:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 04474b64 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 04474b64 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 04474b64 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2518 (Wed Jul 25 04:20:06 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2519 (Wed Jul 25 04:20:06 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f3fa6dd6", "local_conf_timestamp": 2519, "host-ts": 2518}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2512 (Wed Jul 25 04:19:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2512 (Wed Jul 25 04:19:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "569db239", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true} lago.ssh: DEBUG: start task:697a24f7-bb86-4e63-abc3-0ee588786d0a:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:697a24f7-bb86-4e63-abc3-0ee588786d0a:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 07152f64 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 07152f64 on lago-he-node-ng-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 07152f64 on lago-he-node-ng-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2518 (Wed Jul 25 04:20:06 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2519 (Wed Jul 25 04:20:06 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f3fa6dd6", "local_conf_timestamp": 2519, "host-ts": 2518}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2512 (Wed Jul 25 04:19:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2512 (Wed Jul 25 04:19:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "eng ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f2f900dd140> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/008_restart_he_vm.py", line 165, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/008_restart_he_vm.py", line 106, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) RuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2518 (Wed Jul 25 04:20:06 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2519 (Wed Jul 25 04:20:06 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f3fa6dd6", "local_conf_timestamp": 2519, "host-ts": 2518}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2512 (Wed Jul 25 04:19:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2512 (Wed Jul 25 04:19:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "eng --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/008_restart_he_vm.py", line 49, in restart_he_vm _wait_for_engine_health(host) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/008_restart_he_vm.py", line 163, in _wait_for_engine_health testlib.assert_true_within_long(lambda: any( File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 271, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 258, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/008_restart_he_vm.py", line 165, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/008_restart_he_vm.py", line 106, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) 'could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2518 (Wed Jul 25 04:20:06 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2519 (Wed Jul 25 04:20:06 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f3fa6dd6", "local_conf_timestamp": 2519, "host-ts": 2518}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2512 (Wed Jul 25 04:19:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2512 (Wed Jul 25 04:19:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "eng\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:932dbfdb-8f78-42c6-9b1e-6aadee5d8091:Get ssh client for lago-he-node-ng-suite-master-host-1:\nlago.ssh: DEBUG: end task:932dbfdb-8f78-42c6-9b1e-6aadee5d8091:Get ssh client for lago-he-node-ng-suite-master-host-1:\nlago.ssh: DEBUG: Running 7763a99a on lago-he-node-ng-suite-master-host-1: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 7763a99a on lago-he-node-ng-suite-master-host-1 returned with 0\nlago.ssh: DEBUG: Command 7763a99a on lago-he-node-ng-suite-master-host-1 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2275 (Wed Jul 25 04:16:03 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2275 (Wed Jul 25 04:16:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "634493fe", "local_conf_timestamp": 2275, "host-ts": 2275}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2281 (Wed Jul 25 04:16:08 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2281 (Wed Jul 25 04:16:08 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69ca08d9", "local_conf_timestamp": 2281, "host-ts": 2281}, "global_maintenance": true}\n\nroot: INFO: * Shutting down HE VM on host: lago-he-node-ng-suite-master-host-0\nlago.ssh: DEBUG: start task:2bd52306-1957-43f5-a4cd-a16a4223bbd2:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:2bd52306-1957-43f5-a4cd-a16a4223bbd2:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 78525040 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-shutdown\nlago.ssh: DEBUG: Command 78525040 on lago-he-node-ng-suite-master-host-0 returned with 0\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be down...\nlago.ssh: DEBUG: start task:75e52da9-c741-4089-8bda-708623dbb4cc:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:75e52da9-c741-4089-8bda-708623dbb4cc:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 797baba6 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 797baba6 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 797baba6 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2285 (Wed Jul 25 04:16:13 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2286 (Wed Jul 25 04:16:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "bc8dc98f", "local_conf_timestamp": 2286, "host-ts": 2285}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2281 (Wed Jul 25 04:16:08 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2281 (Wed Jul 25 04:16:08 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69ca08d9", "local_conf_timestamp": 2281, "host-ts": 2281}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ac00a93a-e808-4f72-bb6f-f29c7bc259a6:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:ac00a93a-e808-4f72-bb6f-f29c7bc259a6:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 812bb044 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 812bb044 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 812bb044 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2295 (Wed Jul 25 04:16:22 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2295 (Wed Jul 25 04:16:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf71ee59", "local_conf_timestamp": 2295, "host-ts": 2295}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2291 (Wed Jul 25 04:16:18 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2291 (Wed Jul 25 04:16:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "47e1cb10", "local_conf_timestamp": 2291, "host-ts": 2291}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:bb5c3ebf-f315-4147-9664-83fbdfb0b5e8:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:bb5c3ebf-f315-4147-9664-83fbdfb0b5e8:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 83e14d9e on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 83e14d9e on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 83e14d9e on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2305 (Wed Jul 25 04:16:33 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2305 (Wed Jul 25 04:16:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "16c42d10", "local_conf_timestamp": 2305, "host-ts": 2305}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2301 (Wed Jul 25 04:16:28 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2301 (Wed Jul 25 04:16:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "21734107", "local_conf_timestamp": 2301, "host-ts": 2301}, "global_maintenance": true}\n\nroot: INFO: * VM is down.\nroot: INFO: * Stopping services...\nlago.ssh: DEBUG: start task:a91ce735-904b-4589-8d49-374d4d4e5233:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:a91ce735-904b-4589-8d49-374d4d4e5233:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 84cdbff8 on lago-he-node-ng-suite-master-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command 84cdbff8 on lago-he-node-ng-suite-master-host-0 returned with 0\nroot: INFO: * Starting services...\nlago.ssh: DEBUG: start task:69e2b63b-31fc-464e-b7ff-c499a95be86e:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:69e2b63b-31fc-464e-b7ff-c499a95be86e:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 8dac9a18 on lago-he-node-ng-suite-master-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command 8dac9a18 on lago-he-node-ng-suite-master-host-0 returned with 0\nroot: INFO: * Waiting for agent to be ready...\nlago.ssh: DEBUG: start task:5c41febb-2a25-4889-a7c1-5c9da7a33f96:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:5c41febb-2a25-4889-a7c1-5c9da7a33f96:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 909bffb6 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 909bffb6 on lago-he-node-ng-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 909bffb6 on lago-he-node-ng-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:56d0e0c8-3a30-4b6e-acf5-ba1e387ceb7c:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:56d0e0c8-3a30-4b6e-acf5-ba1e387ceb7c:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 977e0a9a on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 977e0a9a on lago-he-node-ng-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 977e0a9a on lago-he-node-ng-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:4b36c31c-c6b7-4bbb-b038-b67e2a26e788:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:4b36c31c-c6b7-4bbb-b038-b67e2a26e788:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 9a1513fc on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 9a1513fc on lago-he-node-ng-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 9a1513fc on lago-he-node-ng-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:e3e68036-c86b-4a3e-ad6d-788272025c42:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:e3e68036-c86b-4a3e-ad6d-788272025c42:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 9cbc9a94 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 9cbc9a94 on lago-he-node-ng-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 9cbc9a94 on lago-he-node-ng-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:7b9f8a19-b7dc-4d00-9202-e07d88888dfe:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:7b9f8a19-b7dc-4d00-9202-e07d88888dfe:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 9f7865c4 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 9f7865c4 on lago-he-node-ng-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 9f7865c4 on lago-he-node-ng-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:1944f900-1947-46ca-b701-3f71d3025a23:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:1944f900-1947-46ca-b701-3f71d3025a23:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running a212e07a on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command a212e07a on lago-he-node-ng-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command a212e07a on lago-he-node-ng-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:289afe1e-8a48-4e1f-b585-c7afe283cc2a:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:289afe1e-8a48-4e1f-b585-c7afe283cc2a:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running a4b92d66 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command a4b92d66 on lago-he-node-ng-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command a4b92d66 on lago-he-node-ng-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:65b385d4-2abe-4c8a-9f9f-37904c3320b5:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:65b385d4-2abe-4c8a-9f9f-37904c3320b5:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running a755a3f6 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command a755a3f6 on lago-he-node-ng-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command a755a3f6 on lago-he-node-ng-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:179389d5-7a94-4742-9328-73c41ed07ed0:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:179389d5-7a94-4742-9328-73c41ed07ed0:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running a9fe6f48 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command a9fe6f48 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a9fe6f48 on lago-he-node-ng-suite-master-host-0 output:\n \n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\n\n--== Host lago-he-node-ng-suite-master-host-0 (id: 1) status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : True\nHostname : lago-he-node-ng-suite-master-host-0\nHost ID : 1\nEngine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}\nScore : 0\nstopped : False\nLocal maintenance : False\ncrc32 : 744e472a\nlocal_conf_timestamp : 2368\nHost timestamp : 2367\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2367 (Wed Jul 25 04:17:35 2018)\n\thost-id=1\n\tscore=0\n\tvm_conf_refresh_time=2368 (Wed Jul 25 04:17:35 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=ReinitializeFSM\n\tstopped=False\n\n\n--== Host lago-he-node-ng-suite-master-host-1 (id: 2) status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : False\nHostname : lago-he-node-ng-suite-master-host-1\nHost ID : 2\nEngine status : unknown stale-data\nScore : 3400\nstopped : False\nLocal maintenance : False\ncrc32 : c58bc9b1\nlocal_conf_timestamp : 2361\nHost timestamp : 2361\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2361 (Wed Jul 25 04:17:28 2018)\n\thost-id=2\n\tscore=3400\n\tvm_conf_refresh_time=2361 (Wed Jul 25 04:17:29 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=GlobalMaintenance\n\tstopped=False\n\n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\nroot: INFO: * Agent is ready.\nroot: INFO: * Starting VM...\nlago.ssh: DEBUG: start task:771d13e3-aa83-4b0c-ad2f-73254bee44b5:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:771d13e3-aa83-4b0c-ad2f-73254bee44b5:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running ab02495a on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-start\nlago.ssh: DEBUG: Command ab02495a on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ab02495a on lago-he-node-ng-suite-master-host-0 output:\n VM exists and is down, cleaning up and restarting\n\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be UP...\nlago.ssh: DEBUG: start task:9b23a87d-00d7-4b1b-9df3-d4f2a93a347a:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:9b23a87d-00d7-4b1b-9df3-d4f2a93a347a:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running ad1c960a on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ad1c960a on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ad1c960a on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2367 (Wed Jul 25 04:17:35 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=2368 (Wed Jul 25 04:17:35 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=ReinitializeFSM\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "744e472a", "local_conf_timestamp": 2368, "host-ts": 2367}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2361 (Wed Jul 25 04:17:28 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2361 (Wed Jul 25 04:17:29 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c58bc9b1", "local_conf_timestamp": 2361, "host-ts": 2361}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:38723e85-fc65-4a4c-903e-49754f691574:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:38723e85-fc65-4a4c-903e-49754f691574:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running b40283bc on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b40283bc on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command b40283bc on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2377 (Wed Jul 25 04:17:44 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2377 (Wed Jul 25 04:17:45 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "40391805", "local_conf_timestamp": 2377, "host-ts": 2377}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2381 (Wed Jul 25 04:17:48 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2381 (Wed Jul 25 04:17:49 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8b5c590e", "local_conf_timestamp": 2381, "host-ts": 2381}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:cd1092f6-e8c6-40ee-99d2-6a6edd7be1b5:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:cd1092f6-e8c6-40ee-99d2-6a6edd7be1b5:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running b6c56286 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b6c56286 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command b6c56286 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2387 (Wed Jul 25 04:17:55 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2388 (Wed Jul 25 04:17:55 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d2ce9013", "local_conf_timestamp": 2388, "host-ts": 2387}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2381 (Wed Jul 25 04:17:48 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2381 (Wed Jul 25 04:17:49 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8b5c590e", "local_conf_timestamp": 2381, "host-ts": 2381}, "global_maintenance": true}\n\nroot: INFO: * VM is UP.\nroot: INFO: * Waiting for engine to start...\nlago.ssh: DEBUG: start task:567f9428-43b1-44f5-ad18-0f4a01c91b13:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:567f9428-43b1-44f5-ad18-0f4a01c91b13:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running b7c81b4c on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b7c81b4c on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command b7c81b4c on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2387 (Wed Jul 25 04:17:55 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2388 (Wed Jul 25 04:17:55 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d2ce9013", "local_conf_timestamp": 2388, "host-ts": 2387}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2381 (Wed Jul 25 04:17:48 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2381 (Wed Jul 25 04:17:49 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8b5c590e", "local_conf_timestamp": 2381, "host-ts": 2381}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:029af7ab-0480-465a-ad7d-9d3a37b5b355:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:029af7ab-0480-465a-ad7d-9d3a37b5b355:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running bec6f814 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command bec6f814 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command bec6f814 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2398 (Wed Jul 25 04:18:05 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2398 (Wed Jul 25 04:18:06 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "412e4ba9", "local_conf_timestamp": 2398, "host-ts": 2398}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2391 (Wed Jul 25 04:17:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2392 (Wed Jul 25 04:17:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "3c7ecb7d", "local_conf_timestamp": 2392, "host-ts": 2391}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:13363bf6-bedc-4e09-814b-bf03d9839892:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:13363bf6-bedc-4e09-814b-bf03d9839892:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running c1a71ce4 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command c1a71ce4 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command c1a71ce4 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2408 (Wed Jul 25 04:18:16 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2408 (Wed Jul 25 04:18:16 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9432151", "local_conf_timestamp": 2408, "host-ts": 2408}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2402 (Wed Jul 25 04:18:09 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2402 (Wed Jul 25 04:18:09 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9a202027", "local_conf_timestamp": 2402, "host-ts": 2402}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8e86ea8a-ed97-49c2-aa31-0d07662c480f:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:8e86ea8a-ed97-49c2-aa31-0d07662c480f:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running c4dffe62 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command c4dffe62 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command c4dffe62 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2408 (Wed Jul 25 04:18:16 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2408 (Wed Jul 25 04:18:16 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9432151", "local_conf_timestamp": 2408, "host-ts": 2408}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2402 (Wed Jul 25 04:18:09 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2402 (Wed Jul 25 04:18:09 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9a202027", "local_conf_timestamp": 2402, "host-ts": 2402}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:e801a931-3f5c-463c-b00e-07905169e39b:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:e801a931-3f5c-463c-b00e-07905169e39b:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running c86e9048 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command c86e9048 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command c86e9048 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2418 (Wed Jul 25 04:18:26 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2419 (Wed Jul 25 04:18:26 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "625c3f40", "local_conf_timestamp": 2419, "host-ts": 2418}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2412 (Wed Jul 25 04:18:20 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2413 (Wed Jul 25 04:18:20 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c30cd378", "local_conf_timestamp": 2413, "host-ts": 2412}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:fc9d9c05-cc70-49f4-9195-5801981130da:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:fc9d9c05-cc70-49f4-9195-5801981130da:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running cb60cc6c on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command cb60cc6c on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command cb60cc6c on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2418 (Wed Jul 25 04:18:26 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2419 (Wed Jul 25 04:18:26 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "625c3f40", "local_conf_timestamp": 2419, "host-ts": 2418}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2412 (Wed Jul 25 04:18:20 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2413 (Wed Jul 25 04:18:20 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c30cd378", "local_conf_timestamp": 2413, "host-ts": 2412}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:a08eb28b-2662-4ab3-a80b-77ef2581a073:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:a08eb28b-2662-4ab3-a80b-77ef2581a073:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running cee93edc on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command cee93edc on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command cee93edc on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2428 (Wed Jul 25 04:18:35 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2428 (Wed Jul 25 04:18:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ace23939", "local_conf_timestamp": 2428, "host-ts": 2428}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2422 (Wed Jul 25 04:18:29 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2422 (Wed Jul 25 04:18:29 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c677a7b5", "local_conf_timestamp": 2422, "host-ts": 2422}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:5475c36c-7873-4bb4-b0a4-89b7a176ed10:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:5475c36c-7873-4bb4-b0a4-89b7a176ed10:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running d2517044 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command d2517044 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command d2517044 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2438 (Wed Jul 25 04:18:46 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2439 (Wed Jul 25 04:18:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b02c5c74", "local_conf_timestamp": 2439, "host-ts": 2438}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2432 (Wed Jul 25 04:18:39 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2432 (Wed Jul 25 04:18:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e85c647c", "local_conf_timestamp": 2432, "host-ts": 2432}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:378598d4-4368-44ff-b912-b640c5a049b0:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:378598d4-4368-44ff-b912-b640c5a049b0:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running d56c33a4 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command d56c33a4 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command d56c33a4 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2438 (Wed Jul 25 04:18:46 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2439 (Wed Jul 25 04:18:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b02c5c74", "local_conf_timestamp": 2439, "host-ts": 2438}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2432 (Wed Jul 25 04:18:39 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2432 (Wed Jul 25 04:18:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e85c647c", "local_conf_timestamp": 2432, "host-ts": 2432}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:7f5db09d-6b15-4582-8818-3c99d0291ce5:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:7f5db09d-6b15-4582-8818-3c99d0291ce5:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running d863856c on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command d863856c on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command d863856c on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2448 (Wed Jul 25 04:18:55 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2448 (Wed Jul 25 04:18:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "0f24c3a2", "local_conf_timestamp": 2448, "host-ts": 2448}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2442 (Wed Jul 25 04:18:49 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2442 (Wed Jul 25 04:18:49 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "228f2f03", "local_conf_timestamp": 2442, "host-ts": 2442}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:639271a3-0803-4020-966c-200e8adb5117:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:639271a3-0803-4020-966c-200e8adb5117:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running db6b7cf6 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command db6b7cf6 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command db6b7cf6 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2448 (Wed Jul 25 04:18:55 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2448 (Wed Jul 25 04:18:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "0f24c3a2", "local_conf_timestamp": 2448, "host-ts": 2448}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2442 (Wed Jul 25 04:18:49 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2442 (Wed Jul 25 04:18:49 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "228f2f03", "local_conf_timestamp": 2442, "host-ts": 2442}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:d1c10391-bcf5-4501-ae6f-47a95d795283:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:d1c10391-bcf5-4501-ae6f-47a95d795283:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running de65a36e on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command de65a36e on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command de65a36e on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2458 (Wed Jul 25 04:19:06 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2458 (Wed Jul 25 04:19:06 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "64ed9679", "local_conf_timestamp": 2458, "host-ts": 2458}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2452 (Wed Jul 25 04:18:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2452 (Wed Jul 25 04:18:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "0ca4ecca", "local_conf_timestamp": 2452, "host-ts": 2452}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:4e03db00-d1d4-43b0-bcfd-a21d3c32b7a7:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:4e03db00-d1d4-43b0-bcfd-a21d3c32b7a7:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running e173ed40 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e173ed40 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command e173ed40 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2458 (Wed Jul 25 04:19:06 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2458 (Wed Jul 25 04:19:06 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "64ed9679", "local_conf_timestamp": 2458, "host-ts": 2458}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2452 (Wed Jul 25 04:18:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2452 (Wed Jul 25 04:18:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "0ca4ecca", "local_conf_timestamp": 2452, "host-ts": 2452}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:d8173e8d-c1e7-49d4-8863-b1007ff2704f:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:d8173e8d-c1e7-49d4-8863-b1007ff2704f:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running e462c878 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e462c878 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command e462c878 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2468 (Wed Jul 25 04:19:15 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2468 (Wed Jul 25 04:19:15 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d8b66814", "local_conf_timestamp": 2468, "host-ts": 2468}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2462 (Wed Jul 25 04:19:10 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2463 (Wed Jul 25 04:19:10 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "09df9807", "local_conf_timestamp": 2463, "host-ts": 2462}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:27e25ae4-3414-4640-8869-9cffad8f6814:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:27e25ae4-3414-4640-8869-9cffad8f6814:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running e794bfec on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e794bfec on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command e794bfec on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2468 (Wed Jul 25 04:19:15 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2468 (Wed Jul 25 04:19:15 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d8b66814", "local_conf_timestamp": 2468, "host-ts": 2468}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2462 (Wed Jul 25 04:19:10 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2463 (Wed Jul 25 04:19:10 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "09df9807", "local_conf_timestamp": 2463, "host-ts": 2462}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:d23aa18c-4462-4239-8db5-c29feecac71e:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:d23aa18c-4462-4239-8db5-c29feecac71e:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running eb998bd6 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command eb998bd6 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command eb998bd6 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2478 (Wed Jul 25 04:19:26 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2479 (Wed Jul 25 04:19:26 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "41565544", "local_conf_timestamp": 2479, "host-ts": 2478}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2472 (Wed Jul 25 04:19:19 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2472 (Wed Jul 25 04:19:20 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "50f36b58", "local_conf_timestamp": 2472, "host-ts": 2472}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:37e54f8a-9fe3-47cd-8896-bb560cc5e4d1:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:37e54f8a-9fe3-47cd-8896-bb560cc5e4d1:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running eed21502 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command eed21502 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command eed21502 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2478 (Wed Jul 25 04:19:26 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2479 (Wed Jul 25 04:19:26 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "41565544", "local_conf_timestamp": 2479, "host-ts": 2478}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2472 (Wed Jul 25 04:19:19 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2472 (Wed Jul 25 04:19:20 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "50f36b58", "local_conf_timestamp": 2472, "host-ts": 2472}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:b6793d51-1124-40aa-9f42-a2107d7649cc:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:b6793d51-1124-40aa-9f42-a2107d7649cc:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running f25fec4e on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f25fec4e on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command f25fec4e on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2488 (Wed Jul 25 04:19:35 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2488 (Wed Jul 25 04:19:35 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "16f668c8", "local_conf_timestamp": 2488, "host-ts": 2488}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2482 (Wed Jul 25 04:19:30 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2483 (Wed Jul 25 04:19:30 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "470808b8", "local_conf_timestamp": 2483, "host-ts": 2482}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:6fb40f9a-5636-47d5-a5e7-cf1835987996:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:6fb40f9a-5636-47d5-a5e7-cf1835987996:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running f5dc5560 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f5dc5560 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command f5dc5560 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2498 (Wed Jul 25 04:19:46 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2499 (Wed Jul 25 04:19:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "0a380d85", "local_conf_timestamp": 2499, "host-ts": 2498}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2492 (Wed Jul 25 04:19:39 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2492 (Wed Jul 25 04:19:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1e24fbe7", "local_conf_timestamp": 2492, "host-ts": 2492}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:54953ae8-9f55-4cb1-bb4c-7ec68dc9879f:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:54953ae8-9f55-4cb1-bb4c-7ec68dc9879f:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running f99d7d32 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f99d7d32 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command f99d7d32 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2498 (Wed Jul 25 04:19:46 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2499 (Wed Jul 25 04:19:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "0a380d85", "local_conf_timestamp": 2499, "host-ts": 2498}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2492 (Wed Jul 25 04:19:39 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2492 (Wed Jul 25 04:19:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1e24fbe7", "local_conf_timestamp": 2492, "host-ts": 2492}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:2d207ff4-7778-45e7-abe3-56af39bec1c8:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:2d207ff4-7778-45e7-abe3-56af39bec1c8:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running fd26f32a on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fd26f32a on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command fd26f32a on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2508 (Wed Jul 25 04:19:55 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2508 (Wed Jul 25 04:19:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ef34089b", "local_conf_timestamp": 2508, "host-ts": 2508}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2502 (Wed Jul 25 04:19:49 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2502 (Wed Jul 25 04:19:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "78b671f0", "local_conf_timestamp": 2502, "host-ts": 2502}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:4819a00d-a93c-4b9d-9439-77204d723a0a:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:4819a00d-a93c-4b9d-9439-77204d723a0a:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 00ce0cca on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 00ce0cca on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 00ce0cca on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2508 (Wed Jul 25 04:19:55 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2508 (Wed Jul 25 04:19:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ef34089b", "local_conf_timestamp": 2508, "host-ts": 2508}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2502 (Wed Jul 25 04:19:49 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2502 (Wed Jul 25 04:19:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "78b671f0", "local_conf_timestamp": 2502, "host-ts": 2502}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:2eadab93-166a-4267-8a57-0d457b253235:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:2eadab93-166a-4267-8a57-0d457b253235:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 04474b64 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 04474b64 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 04474b64 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2518 (Wed Jul 25 04:20:06 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2519 (Wed Jul 25 04:20:06 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f3fa6dd6", "local_conf_timestamp": 2519, "host-ts": 2518}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2512 (Wed Jul 25 04:19:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2512 (Wed Jul 25 04:19:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "569db239", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:697a24f7-bb86-4e63-abc3-0ee588786d0a:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:697a24f7-bb86-4e63-abc3-0ee588786d0a:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 07152f64 on lago-he-node-ng-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 07152f64 on lago-he-node-ng-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 07152f64 on lago-he-node-ng-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2518 (Wed Jul 25 04:20:06 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2519 (Wed Jul 25 04:20:06 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f3fa6dd6", "local_conf_timestamp": 2519, "host-ts": 2518}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2512 (Wed Jul 25 04:19:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2512 (Wed Jul 25 04:19:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "eng\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f2f900dd140>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/008_restart_he_vm.py", line 165, in <lambda>\n for k, v in _get_he_status(host).items()\n File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/008_restart_he_vm.py", line 106, in _get_he_status\n raise RuntimeError(\'could not parse JSON: %s\' % ret.out)\nRuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2518 (Wed Jul 25 04:20:06 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2519 (Wed Jul 25 04:20:06 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f3fa6dd6", "local_conf_timestamp": 2519, "host-ts": 2518}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2512 (Wed Jul 25 04:19:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2512 (Wed Jul 25 04:19:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-master-host-1", "host-id": 2, "eng\n--------------------- >> end captured logging << ---------------------'

1 1

[JIRA] (OVIRT-2368) Please create some ovirt-ansible-* github repositories
by Barak Korren (oVirt JIRA) 25 Jul '18

25 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2368?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-2368: ------------------------------------- Repos created: * [ovirt-ansible-dump-db|https://github.com/oVirt/ovirt-ansible-dump-db] * [ovirt-ansible-rename-db|https://github.com/oVirt/ovirt-ansible-rename-db] * [ovirt-ansible-remote-db|https://github.com/oVirt/ovirt-ansible-remote-db] * [ovirt-ansible-local-to-remote-db|https://github.com/oVirt/ovirt-ansible-loc…] [~lleistne] please confirm everything works as you requested. > Please create some ovirt-ansible-* github repositories > ------------------------------------------------------ > > Key: OVIRT-2368 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2368 > Project: oVirt - virtualization made easy > Issue Type: Task > Components: GitHub > Reporter: Lucie Leistnerova > Assignee: infra > > Please create this github repositories: > ovirt-ansible-dump-db > ovirt-ansible-rename-db > ovirt-ansible-remote-db > ovirt-ansible-local-to-remote-db > And assign leistnerova as administrator. > Thank you. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2368) Please create some ovirt-ansible-* github repositories
by Barak Korren (oVirt JIRA) 25 Jul '18

25 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2368?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-2368: -------------------------------- Component/s: GitHub > Please create some ovirt-ansible-* github repositories > ------------------------------------------------------ > > Key: OVIRT-2368 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2368 > Project: oVirt - virtualization made easy > Issue Type: Task > Components: GitHub > Reporter: Lucie Leistnerova > Assignee: infra > > Please create this github repositories: > ovirt-ansible-dump-db > ovirt-ansible-rename-db > ovirt-ansible-remote-db > ovirt-ansible-local-to-remote-db > And assign leistnerova as administrator. > Thank you. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2370) Please create ovirt-ansible-shutdown-env github repository
by Barak Korren (oVirt JIRA) 25 Jul '18

25 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2370?page=com.atlassian.jira.… ] Barak Korren reassigned OVIRT-2370: ----------------------------------- Assignee: Barak Korren (was: infra) > Please create ovirt-ansible-shutdown-env github repository > ---------------------------------------------------------- > > Key: OVIRT-2370 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2370 > Project: oVirt - virtualization made easy > Issue Type: Task > Components: GitHub > Reporter: Simone Tiraboschi > Assignee: Barak Korren > > Please create ovirt-ansible-shutdown-env github repository. > Please assign tiraboschi, irosenzw, didib and sandrobonazzola as administrators. > Thanks. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0