[oVirt Jenkins] ovirt-system-tests_he-basic-suite-4.2 - Build # 510 - Failure!
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.2/510/ Build Number: 510 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #510 [Roy Golan] Introduce openshift-on-ovirt ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 006_network_by_label.assign_hosts_network_label Error Message: status: 409 reason: Conflict detail: Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later. -------------------- >> begin captured logging << -------------------- lago.utils: ERROR: Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/006_network_by_label.py", line 57, in _assign_host_network_label host_nic=nic File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 16231, in add headers={"Correlation-Id":correlation_id, "Expect":expect} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add return self.request('POST', url, body, headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) RequestError: status: 409 reason: Conflict detail: Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later. --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 69, in wrapper return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/006_network_by_label.py", line 65, in assign_hosts_network_label nt.assert_true(all(vt.join_all())) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/006_network_by_label.py", line 57, in _assign_host_network_label host_nic=nic File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 16231, in add headers={"Correlation-Id":correlation_id, "Expect":expect} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add return self.request('POST', url, body, headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) '\r\nstatus: 409\r\nreason: Conflict\r\ndetail: Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later.\n-------------------- >> begin captured logging << --------------------\nlago.utils: ERROR: Error while running thread\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue\n queue.put({\'return\': func()})\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/006_network_by_label.py", line 57, in _assign_host_network_label\n host_nic=nic\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 16231, in add\n headers={"Correlation-Id":correlation_id, "Expect":expect}\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add\n return self.request(\'POST\', url, body, headers, cls=cls)\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request\n persistent_auth=self.__persistent_auth\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request\n persistent_auth)\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request\n raise errors.RequestError(response_code, response_reason, response_body)\nRequestError: \r\nstatus: 409\r\nreason: Conflict\r\ndetail: Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later.\n--------------------- >> end captured logging << ---------------------'
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.2/511/ Build Number: 511 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #510 [Roy Golan] Introduce openshift-on-ovirt Changes for Build #511 [Roy Golan] Introduce openshift-on-ovirt ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 008_restart_he_vm.restart_he_vm Error Message: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2274 (Sun Aug 26 14:05:46 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2275 (Sun Aug 26 14:05:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "3a741d0e", "local_conf_timestamp": 2275, "host-ts": 2274}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2266 (Sun Aug 26 14:05:37 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2267 (Sun Aug 26 14:05:38 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "en -------------------- >> begin captured logging << -------------------- lago.ssh: DEBUG: start task:8c4ed11c-bc1c-49c8-8110-8626677a50ca:Get ssh client for lago-he-basic-suite-4-2-host-1: lago.ssh: DEBUG: end task:8c4ed11c-bc1c-49c8-8110-8626677a50ca:Get ssh client for lago-he-basic-suite-4-2-host-1: lago.ssh: DEBUG: Running 4a437212 on lago-he-basic-suite-4-2-host-1: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 4a437212 on lago-he-basic-suite-4-2-host-1 returned with 0 lago.ssh: DEBUG: Command 4a437212 on lago-he-basic-suite-4-2-host-1 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2107 (Sun Aug 26 14:02:58 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2107 (Sun Aug 26 14:02:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fdf121ef", "local_conf_timestamp": 2107, "host-ts": 2107}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2116 (Sun Aug 26 14:03:07 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2117 (Sun Aug 26 14:03:08 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "df7cb02f", "local_conf_timestamp": 2117, "host-ts": 2116}, "global_maintenance": true} root: INFO: * Shutting down HE VM on host: lago-he-basic-suite-4-2-host-0 lago.ssh: DEBUG: start task:344fccf4-6b8e-406d-a5d7-3dcc2bf664a8:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:344fccf4-6b8e-406d-a5d7-3dcc2bf664a8:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 4abd7a3a on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-shutdown lago.ssh: DEBUG: Command 4abd7a3a on lago-he-basic-suite-4-2-host-0 returned with 0 root: INFO: * Command succeeded root: INFO: * Waiting for VM to be down... lago.ssh: DEBUG: start task:8d831cb7-0035-4e95-ae82-bc1c3e136891:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:8d831cb7-0035-4e95-ae82-bc1c3e136891:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 4b534ace on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 4b534ace on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 4b534ace on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2117 (Sun Aug 26 14:03:09 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2118 (Sun Aug 26 14:03:09 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "4623b12e", "local_conf_timestamp": 2118, "host-ts": 2117}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2116 (Sun Aug 26 14:03:07 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2117 (Sun Aug 26 14:03:08 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "df7cb02f", "local_conf_timestamp": 2117, "host-ts": 2116}, "global_maintenance": true} lago.ssh: DEBUG: start task:1098018c-f628-4028-b0c3-9a6a3287f1b1:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:1098018c-f628-4028-b0c3-9a6a3287f1b1:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 51e1c168 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 51e1c168 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 51e1c168 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2127 (Sun Aug 26 14:03:18 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2127 (Sun Aug 26 14:03:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "afa65c75", "local_conf_timestamp": 2127, "host-ts": 2127}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2126 (Sun Aug 26 14:03:17 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2126 (Sun Aug 26 14:03:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9effa9f6", "local_conf_timestamp": 2126, "host-ts": 2126}, "global_maintenance": true} root: INFO: * VM is down. root: INFO: * Stopping services... lago.ssh: DEBUG: start task:3e1b8bd7-4f95-448b-84d0-39468d7d35c4:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:3e1b8bd7-4f95-448b-84d0-39468d7d35c4:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 525853f0 on lago-he-basic-suite-4-2-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command 525853f0 on lago-he-basic-suite-4-2-host-0 returned with 0 root: INFO: * Starting services... lago.ssh: DEBUG: start task:62250dc4-1c20-4a0e-aa33-a70bda976541:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:62250dc4-1c20-4a0e-aa33-a70bda976541:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 5b86f6f2 on lago-he-basic-suite-4-2-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command 5b86f6f2 on lago-he-basic-suite-4-2-host-0 returned with 0 root: INFO: * Waiting for agent to be ready... lago.ssh: DEBUG: start task:02d149e1-b0cf-43c7-aae9-e803d65a7abe:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:02d149e1-b0cf-43c7-aae9-e803d65a7abe:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 5d765b60 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 5d765b60 on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command 5d765b60 on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:4a7a6d4f-1d74-488f-bd83-275c4845e2c3:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:4a7a6d4f-1d74-488f-bd83-275c4845e2c3:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 63cf3e46 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 63cf3e46 on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command 63cf3e46 on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:2e3309d0-b701-4aab-b0ff-e63603602ec8:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:2e3309d0-b701-4aab-b0ff-e63603602ec8:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 65f19bba on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 65f19bba on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command 65f19bba on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:b1c232ea-242a-46db-8678-bdec75eda021:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:b1c232ea-242a-46db-8678-bdec75eda021:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 681c6e60 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 681c6e60 on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command 681c6e60 on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:00a3211c-bc3f-4fe6-bc9b-4674e5def441:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:00a3211c-bc3f-4fe6-bc9b-4674e5def441:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 6a448a06 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 6a448a06 on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command 6a448a06 on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:f074443d-d9c8-41cf-a414-f5afa3dd390c:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:f074443d-d9c8-41cf-a414-f5afa3dd390c:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 6c63a7a4 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 6c63a7a4 on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command 6c63a7a4 on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:44e70a33-7ac8-407e-91d3-703380ea8114:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:44e70a33-7ac8-407e-91d3-703380ea8114:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 6e8069e6 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 6e8069e6 on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command 6e8069e6 on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:ac830e90-8461-4575-bd1e-b35832b7c7bf:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:ac830e90-8461-4575-bd1e-b35832b7c7bf:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 70a0250e on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 70a0250e on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command 70a0250e on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:baa6f0d0-3ea7-4ef1-826e-2b995c888019:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:baa6f0d0-3ea7-4ef1-826e-2b995c888019:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 72c118ac on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 72c118ac on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 72c118ac on lago-he-basic-suite-4-2-host-0 output: !! Cluster is in GLOBAL MAINTENANCE mode !! --== Host 1 status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : lago-he-basic-suite-4-2-host-0.lago.local Host ID : 1 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"} Score : 0 stopped : False Local maintenance : False crc32 : 4c6fdf7c local_conf_timestamp : 2185 Host timestamp : 2184 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2184 (Sun Aug 26 14:04:16 2018) host-id=1 score=0 vm_conf_refresh_time=2185 (Sun Aug 26 14:04:16 2018) conf_on_shared_storage=True maintenance=False state=ReinitializeFSM stopped=False --== Host 2 status ==-- conf_on_shared_storage : True Status up-to-date : False Hostname : lago-he-basic-suite-4-2-host-1 Host ID : 2 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : c573d227 local_conf_timestamp : 2176 Host timestamp : 2176 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2176 (Sun Aug 26 14:04:07 2018) host-id=2 score=3400 vm_conf_refresh_time=2176 (Sun Aug 26 14:04:07 2018) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False !! Cluster is in GLOBAL MAINTENANCE mode !! root: INFO: * Agent is ready. root: INFO: * Starting VM... lago.ssh: DEBUG: start task:8bac5eac-9415-4098-9f66-f6ba3bfe02c4:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:8bac5eac-9415-4098-9f66-f6ba3bfe02c4:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 7355ed88 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-start lago.ssh: DEBUG: Command 7355ed88 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 7355ed88 on lago-he-basic-suite-4-2-host-0 output: VM exists and is down, cleaning up and restarting root: INFO: * Command succeeded root: INFO: * Waiting for VM to be UP... lago.ssh: DEBUG: start task:5fbcc916-a582-4a7b-95ba-3347bf9f46ac:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:5fbcc916-a582-4a7b-95ba-3347bf9f46ac:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 74ab61ea on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 74ab61ea on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 74ab61ea on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2184 (Sun Aug 26 14:04:16 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=2185 (Sun Aug 26 14:04:16 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=ReinitializeFSM\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "4c6fdf7c", "local_conf_timestamp": 2185, "host-ts": 2184}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2176 (Sun Aug 26 14:04:07 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2176 (Sun Aug 26 14:04:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c573d227", "local_conf_timestamp": 2176, "host-ts": 2176}, "global_maintenance": true} lago.ssh: DEBUG: start task:ca8c1e4f-693f-4dd3-a170-817d3eb762f4:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:ca8c1e4f-693f-4dd3-a170-817d3eb762f4:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 7b21f624 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 7b21f624 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 7b21f624 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2194 (Sun Aug 26 14:04:25 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2194 (Sun Aug 26 14:04:26 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d681be2e", "local_conf_timestamp": 2194, "host-ts": 2194}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2186 (Sun Aug 26 14:04:17 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2186 (Sun Aug 26 14:04:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "29e75e54", "local_conf_timestamp": 2186, "host-ts": 2186}, "global_maintenance": true} root: INFO: * VM is UP. root: INFO: * Waiting for engine to start... lago.ssh: DEBUG: start task:745ecf00-06f9-4897-bd8a-0ba33de9867c:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:745ecf00-06f9-4897-bd8a-0ba33de9867c:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 7b9a9e30 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 7b9a9e30 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 7b9a9e30 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2194 (Sun Aug 26 14:04:25 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2194 (Sun Aug 26 14:04:26 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d681be2e", "local_conf_timestamp": 2194, "host-ts": 2194}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2186 (Sun Aug 26 14:04:17 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2186 (Sun Aug 26 14:04:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "29e75e54", "local_conf_timestamp": 2186, "host-ts": 2186}, "global_maintenance": true} lago.ssh: DEBUG: start task:bc151d0d-e429-45ec-b326-c4e57b04f706:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:bc151d0d-e429-45ec-b326-c4e57b04f706:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 8206d8e2 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 8206d8e2 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 8206d8e2 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2204 (Sun Aug 26 14:04:36 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2205 (Sun Aug 26 14:04:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "34bf6954", "local_conf_timestamp": 2205, "host-ts": 2204}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2196 (Sun Aug 26 14:04:27 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2197 (Sun Aug 26 14:04:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "4c9c7607", "local_conf_timestamp": 2197, "host-ts": 2196}, "global_maintenance": true} lago.ssh: DEBUG: start task:8f21eb76-51e0-4706-a017-4b06b617459f:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:8f21eb76-51e0-4706-a017-4b06b617459f:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 845130a2 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 845130a2 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 845130a2 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2214 (Sun Aug 26 14:04:46 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2215 (Sun Aug 26 14:04:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9152fdda", "local_conf_timestamp": 2215, "host-ts": 2214}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2206 (Sun Aug 26 14:04:37 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2206 (Sun Aug 26 14:04:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5c524054", "local_conf_timestamp": 2206, "host-ts": 2206}, "global_maintenance": true} lago.ssh: DEBUG: start task:126ecf00-037c-40ed-84a6-3b75751980dc:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:126ecf00-037c-40ed-84a6-3b75751980dc:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 86951798 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 86951798 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 86951798 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2214 (Sun Aug 26 14:04:46 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2215 (Sun Aug 26 14:04:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9152fdda", "local_conf_timestamp": 2215, "host-ts": 2214}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2206 (Sun Aug 26 14:04:37 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2206 (Sun Aug 26 14:04:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5c524054", "local_conf_timestamp": 2206, "host-ts": 2206}, "global_maintenance": true} lago.ssh: DEBUG: start task:e1f2dd2f-6ec9-4f89-af40-88105a9034e7:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:e1f2dd2f-6ec9-4f89-af40-88105a9034e7:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 891db2a4 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 891db2a4 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 891db2a4 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2214 (Sun Aug 26 14:04:46 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2215 (Sun Aug 26 14:04:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9152fdda", "local_conf_timestamp": 2215, "host-ts": 2214}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2206 (Sun Aug 26 14:04:37 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2206 (Sun Aug 26 14:04:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5c524054", "local_conf_timestamp": 2206, "host-ts": 2206}, "global_maintenance": true} lago.ssh: DEBUG: start task:4c0118ba-3f9f-4e7a-a34e-c86745110252:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:4c0118ba-3f9f-4e7a-a34e-c86745110252:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 8b66f4bc on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 8b66f4bc on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 8b66f4bc on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2224 (Sun Aug 26 14:04:55 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2224 (Sun Aug 26 14:04:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "06c7c260", "local_conf_timestamp": 2224, "host-ts": 2224}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2216 (Sun Aug 26 14:04:47 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2217 (Sun Aug 26 14:04:48 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "39296807", "local_conf_timestamp": 2217, "host-ts": 2216}, "global_maintenance": true} lago.ssh: DEBUG: start task:29c432bf-3e91-4b02-979e-bc4ada4b5e31:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:29c432bf-3e91-4b02-979e-bc4ada4b5e31:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 8db71878 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 8db71878 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 8db71878 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2224 (Sun Aug 26 14:04:55 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2224 (Sun Aug 26 14:04:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "06c7c260", "local_conf_timestamp": 2224, "host-ts": 2224}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2216 (Sun Aug 26 14:04:47 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2217 (Sun Aug 26 14:04:48 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "39296807", "local_conf_timestamp": 2217, "host-ts": 2216}, "global_maintenance": true} lago.ssh: DEBUG: start task:8ca20943-46ec-468f-866b-926e9c3e7ea6:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:8ca20943-46ec-468f-866b-926e9c3e7ea6:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 90019b26 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 90019b26 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 90019b26 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2234 (Sun Aug 26 14:05:05 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2234 (Sun Aug 26 14:05:05 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "14bbdaa5", "local_conf_timestamp": 2234, "host-ts": 2234}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2226 (Sun Aug 26 14:04:57 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2226 (Sun Aug 26 14:04:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "78aa71de", "local_conf_timestamp": 2226, "host-ts": 2226}, "global_maintenance": true} lago.ssh: DEBUG: start task:8fca79b1-660e-49e2-83b8-3fc306aa140c:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:8fca79b1-660e-49e2-83b8-3fc306aa140c:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 9269ceba on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9269ceba on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 9269ceba on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2234 (Sun Aug 26 14:05:05 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2234 (Sun Aug 26 14:05:05 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "14bbdaa5", "local_conf_timestamp": 2234, "host-ts": 2234}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2226 (Sun Aug 26 14:04:57 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2226 (Sun Aug 26 14:04:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "78aa71de", "local_conf_timestamp": 2226, "host-ts": 2226}, "global_maintenance": true} lago.ssh: DEBUG: start task:758d9896-bd87-4dbf-8748-35e94f75dd00:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:758d9896-bd87-4dbf-8748-35e94f75dd00:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 94aa1d2e on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 94aa1d2e on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 94aa1d2e on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2234 (Sun Aug 26 14:05:05 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2234 (Sun Aug 26 14:05:05 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "14bbdaa5", "local_conf_timestamp": 2234, "host-ts": 2234}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2226 (Sun Aug 26 14:04:57 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2226 (Sun Aug 26 14:04:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "78aa71de", "local_conf_timestamp": 2226, "host-ts": 2226}, "global_maintenance": true} lago.ssh: DEBUG: start task:360edee2-2fd9-4382-8331-03ae68226f1b:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:360edee2-2fd9-4382-8331-03ae68226f1b:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 971268be on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 971268be on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 971268be on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2244 (Sun Aug 26 14:05:16 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2245 (Sun Aug 26 14:05:16 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1cc8a068", "local_conf_timestamp": 2245, "host-ts": 2244}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2236 (Sun Aug 26 14:05:07 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2236 (Sun Aug 26 14:05:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6ad6691b", "local_conf_timestamp": 2236, "host-ts": 2236}, "global_maintenance": true} lago.ssh: DEBUG: start task:7456e721-7a0a-46da-bbb0-770bbc09ac54:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:7456e721-7a0a-46da-bbb0-770bbc09ac54:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 997f8ad2 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 997f8ad2 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 997f8ad2 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2244 (Sun Aug 26 14:05:16 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2245 (Sun Aug 26 14:05:16 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1cc8a068", "local_conf_timestamp": 2245, "host-ts": 2244}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2236 (Sun Aug 26 14:05:07 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2236 (Sun Aug 26 14:05:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6ad6691b", "local_conf_timestamp": 2236, "host-ts": 2236}, "global_maintenance": true} lago.ssh: DEBUG: start task:536fbae0-0bd0-478e-a20e-325a685274fa:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:536fbae0-0bd0-478e-a20e-325a685274fa:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 9be04cf8 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9be04cf8 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 9be04cf8 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2244 (Sun Aug 26 14:05:16 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2245 (Sun Aug 26 14:05:16 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1cc8a068", "local_conf_timestamp": 2245, "host-ts": 2244}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2236 (Sun Aug 26 14:05:07 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2236 (Sun Aug 26 14:05:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6ad6691b", "local_conf_timestamp": 2236, "host-ts": 2236}, "global_maintenance": true} lago.ssh: DEBUG: start task:e916b2dc-2d57-4571-83ea-a03b5c863fb9:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:e916b2dc-2d57-4571-83ea-a03b5c863fb9:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 9e63cc5c on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9e63cc5c on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 9e63cc5c on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2254 (Sun Aug 26 14:05:25 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2255 (Sun Aug 26 14:05:26 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1e8c2c84", "local_conf_timestamp": 2255, "host-ts": 2254}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2246 (Sun Aug 26 14:05:17 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2247 (Sun Aug 26 14:05:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "62a513d6", "local_conf_timestamp": 2247, "host-ts": 2246}, "global_maintenance": true} lago.ssh: DEBUG: start task:361ba153-6e78-49de-990c-9f8461794e77:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:361ba153-6e78-49de-990c-9f8461794e77:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running a0e98318 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a0e98318 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command a0e98318 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2254 (Sun Aug 26 14:05:25 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2255 (Sun Aug 26 14:05:26 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1e8c2c84", "local_conf_timestamp": 2255, "host-ts": 2254}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2246 (Sun Aug 26 14:05:17 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2247 (Sun Aug 26 14:05:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "62a513d6", "local_conf_timestamp": 2247, "host-ts": 2246}, "global_maintenance": true} lago.ssh: DEBUG: start task:ad5a7a91-e1ae-47a9-bee1-80971fcc5f22:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:ad5a7a91-e1ae-47a9-bee1-80971fcc5f22:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running a354845e on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a354845e on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command a354845e on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2264 (Sun Aug 26 14:05:36 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2265 (Sun Aug 26 14:05:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "280805cb", "local_conf_timestamp": 2265, "host-ts": 2264}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2256 (Sun Aug 26 14:05:27 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2256 (Sun Aug 26 14:05:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "07de3b85", "local_conf_timestamp": 2256, "host-ts": 2256}, "global_maintenance": true} lago.ssh: DEBUG: start task:b5e00fb8-336a-4bc8-82d1-7ab62556304b:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:b5e00fb8-336a-4bc8-82d1-7ab62556304b:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running a5a62c44 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a5a62c44 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command a5a62c44 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2264 (Sun Aug 26 14:05:36 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2265 (Sun Aug 26 14:05:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "280805cb", "local_conf_timestamp": 2265, "host-ts": 2264}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2256 (Sun Aug 26 14:05:27 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2256 (Sun Aug 26 14:05:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "07de3b85", "local_conf_timestamp": 2256, "host-ts": 2256}, "global_maintenance": true} lago.ssh: DEBUG: start task:c942e768-03b4-4e0a-b5a1-8f9bee1f1541:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:c942e768-03b4-4e0a-b5a1-8f9bee1f1541:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running a7ff984a on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a7ff984a on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command a7ff984a on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2264 (Sun Aug 26 14:05:36 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2265 (Sun Aug 26 14:05:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "280805cb", "local_conf_timestamp": 2265, "host-ts": 2264}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2256 (Sun Aug 26 14:05:27 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2256 (Sun Aug 26 14:05:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "07de3b85", "local_conf_timestamp": 2256, "host-ts": 2256}, "global_maintenance": true} lago.ssh: DEBUG: start task:318bc110-f27d-496e-b518-863d03725adf:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:318bc110-f27d-496e-b518-863d03725adf:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running aa43905c on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command aa43905c on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command aa43905c on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2274 (Sun Aug 26 14:05:46 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2275 (Sun Aug 26 14:05:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "3a741d0e", "local_conf_timestamp": 2275, "host-ts": 2274}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2266 (Sun Aug 26 14:05:37 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2267 (Sun Aug 26 14:05:38 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "en ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f3c24244cf8> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 187, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) RuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2274 (Sun Aug 26 14:05:46 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2275 (Sun Aug 26 14:05:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "3a741d0e", "local_conf_timestamp": 2275, "host-ts": 2274}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2266 (Sun Aug 26 14:05:37 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2267 (Sun Aug 26 14:05:38 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "en --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 53, in restart_he_vm _wait_for_engine_health(host) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 185, in _wait_for_engine_health testlib.assert_true_within_long(lambda: any( File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 187, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) 'could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2274 (Sun Aug 26 14:05:46 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2275 (Sun Aug 26 14:05:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "3a741d0e", "local_conf_timestamp": 2275, "host-ts": 2274}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2266 (Sun Aug 26 14:05:37 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2267 (Sun Aug 26 14:05:38 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "en\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:8c4ed11c-bc1c-49c8-8110-8626677a50ca:Get ssh client for lago-he-basic-suite-4-2-host-1:\nlago.ssh: DEBUG: end task:8c4ed11c-bc1c-49c8-8110-8626677a50ca:Get ssh client for lago-he-basic-suite-4-2-host-1:\nlago.ssh: DEBUG: Running 4a437212 on lago-he-basic-suite-4-2-host-1: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 4a437212 on lago-he-basic-suite-4-2-host-1 returned with 0\nlago.ssh: DEBUG: Command 4a437212 on lago-he-basic-suite-4-2-host-1 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2107 (Sun Aug 26 14:02:58 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2107 (Sun Aug 26 14:02:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fdf121ef", "local_conf_timestamp": 2107, "host-ts": 2107}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2116 (Sun Aug 26 14:03:07 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2117 (Sun Aug 26 14:03:08 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "df7cb02f", "local_conf_timestamp": 2117, "host-ts": 2116}, "global_maintenance": true}\n\nroot: INFO: * Shutting down HE VM on host: lago-he-basic-suite-4-2-host-0\nlago.ssh: DEBUG: start task:344fccf4-6b8e-406d-a5d7-3dcc2bf664a8:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:344fccf4-6b8e-406d-a5d7-3dcc2bf664a8:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 4abd7a3a on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-shutdown\nlago.ssh: DEBUG: Command 4abd7a3a on lago-he-basic-suite-4-2-host-0 returned with 0\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be down...\nlago.ssh: DEBUG: start task:8d831cb7-0035-4e95-ae82-bc1c3e136891:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:8d831cb7-0035-4e95-ae82-bc1c3e136891:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 4b534ace on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 4b534ace on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 4b534ace on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2117 (Sun Aug 26 14:03:09 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2118 (Sun Aug 26 14:03:09 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "4623b12e", "local_conf_timestamp": 2118, "host-ts": 2117}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2116 (Sun Aug 26 14:03:07 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2117 (Sun Aug 26 14:03:08 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "df7cb02f", "local_conf_timestamp": 2117, "host-ts": 2116}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:1098018c-f628-4028-b0c3-9a6a3287f1b1:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:1098018c-f628-4028-b0c3-9a6a3287f1b1:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 51e1c168 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 51e1c168 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 51e1c168 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2127 (Sun Aug 26 14:03:18 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2127 (Sun Aug 26 14:03:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "afa65c75", "local_conf_timestamp": 2127, "host-ts": 2127}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2126 (Sun Aug 26 14:03:17 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2126 (Sun Aug 26 14:03:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9effa9f6", "local_conf_timestamp": 2126, "host-ts": 2126}, "global_maintenance": true}\n\nroot: INFO: * VM is down.\nroot: INFO: * Stopping services...\nlago.ssh: DEBUG: start task:3e1b8bd7-4f95-448b-84d0-39468d7d35c4:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:3e1b8bd7-4f95-448b-84d0-39468d7d35c4:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 525853f0 on lago-he-basic-suite-4-2-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command 525853f0 on lago-he-basic-suite-4-2-host-0 returned with 0\nroot: INFO: * Starting services...\nlago.ssh: DEBUG: start task:62250dc4-1c20-4a0e-aa33-a70bda976541:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:62250dc4-1c20-4a0e-aa33-a70bda976541:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 5b86f6f2 on lago-he-basic-suite-4-2-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command 5b86f6f2 on lago-he-basic-suite-4-2-host-0 returned with 0\nroot: INFO: * Waiting for agent to be ready...\nlago.ssh: DEBUG: start task:02d149e1-b0cf-43c7-aae9-e803d65a7abe:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:02d149e1-b0cf-43c7-aae9-e803d65a7abe:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 5d765b60 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 5d765b60 on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command 5d765b60 on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:4a7a6d4f-1d74-488f-bd83-275c4845e2c3:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:4a7a6d4f-1d74-488f-bd83-275c4845e2c3:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 63cf3e46 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 63cf3e46 on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command 63cf3e46 on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:2e3309d0-b701-4aab-b0ff-e63603602ec8:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:2e3309d0-b701-4aab-b0ff-e63603602ec8:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 65f19bba on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 65f19bba on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command 65f19bba on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:b1c232ea-242a-46db-8678-bdec75eda021:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:b1c232ea-242a-46db-8678-bdec75eda021:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 681c6e60 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 681c6e60 on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command 681c6e60 on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:00a3211c-bc3f-4fe6-bc9b-4674e5def441:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:00a3211c-bc3f-4fe6-bc9b-4674e5def441:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 6a448a06 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 6a448a06 on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command 6a448a06 on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:f074443d-d9c8-41cf-a414-f5afa3dd390c:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:f074443d-d9c8-41cf-a414-f5afa3dd390c:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 6c63a7a4 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 6c63a7a4 on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command 6c63a7a4 on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:44e70a33-7ac8-407e-91d3-703380ea8114:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:44e70a33-7ac8-407e-91d3-703380ea8114:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 6e8069e6 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 6e8069e6 on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command 6e8069e6 on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:ac830e90-8461-4575-bd1e-b35832b7c7bf:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:ac830e90-8461-4575-bd1e-b35832b7c7bf:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 70a0250e on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 70a0250e on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command 70a0250e on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:baa6f0d0-3ea7-4ef1-826e-2b995c888019:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:baa6f0d0-3ea7-4ef1-826e-2b995c888019:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 72c118ac on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 72c118ac on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 72c118ac on lago-he-basic-suite-4-2-host-0 output:\n \n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\n\n--== Host 1 status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : True\nHostname : lago-he-basic-suite-4-2-host-0.lago.local\nHost ID : 1\nEngine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}\nScore : 0\nstopped : False\nLocal maintenance : False\ncrc32 : 4c6fdf7c\nlocal_conf_timestamp : 2185\nHost timestamp : 2184\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2184 (Sun Aug 26 14:04:16 2018)\n\thost-id=1\n\tscore=0\n\tvm_conf_refresh_time=2185 (Sun Aug 26 14:04:16 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=ReinitializeFSM\n\tstopped=False\n\n\n--== Host 2 status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : False\nHostname : lago-he-basic-suite-4-2-host-1\nHost ID : 2\nEngine status : unknown stale-data\nScore : 3400\nstopped : False\nLocal maintenance : False\ncrc32 : c573d227\nlocal_conf_timestamp : 2176\nHost timestamp : 2176\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2176 (Sun Aug 26 14:04:07 2018)\n\thost-id=2\n\tscore=3400\n\tvm_conf_refresh_time=2176 (Sun Aug 26 14:04:07 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=GlobalMaintenance\n\tstopped=False\n\n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\nroot: INFO: * Agent is ready.\nroot: INFO: * Starting VM...\nlago.ssh: DEBUG: start task:8bac5eac-9415-4098-9f66-f6ba3bfe02c4:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:8bac5eac-9415-4098-9f66-f6ba3bfe02c4:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 7355ed88 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-start\nlago.ssh: DEBUG: Command 7355ed88 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 7355ed88 on lago-he-basic-suite-4-2-host-0 output:\n VM exists and is down, cleaning up and restarting\n\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be UP...\nlago.ssh: DEBUG: start task:5fbcc916-a582-4a7b-95ba-3347bf9f46ac:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:5fbcc916-a582-4a7b-95ba-3347bf9f46ac:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 74ab61ea on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 74ab61ea on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 74ab61ea on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2184 (Sun Aug 26 14:04:16 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=2185 (Sun Aug 26 14:04:16 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=ReinitializeFSM\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "4c6fdf7c", "local_conf_timestamp": 2185, "host-ts": 2184}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2176 (Sun Aug 26 14:04:07 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2176 (Sun Aug 26 14:04:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c573d227", "local_conf_timestamp": 2176, "host-ts": 2176}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ca8c1e4f-693f-4dd3-a170-817d3eb762f4:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:ca8c1e4f-693f-4dd3-a170-817d3eb762f4:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 7b21f624 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 7b21f624 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 7b21f624 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2194 (Sun Aug 26 14:04:25 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2194 (Sun Aug 26 14:04:26 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d681be2e", "local_conf_timestamp": 2194, "host-ts": 2194}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2186 (Sun Aug 26 14:04:17 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2186 (Sun Aug 26 14:04:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "29e75e54", "local_conf_timestamp": 2186, "host-ts": 2186}, "global_maintenance": true}\n\nroot: INFO: * VM is UP.\nroot: INFO: * Waiting for engine to start...\nlago.ssh: DEBUG: start task:745ecf00-06f9-4897-bd8a-0ba33de9867c:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:745ecf00-06f9-4897-bd8a-0ba33de9867c:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 7b9a9e30 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 7b9a9e30 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 7b9a9e30 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2194 (Sun Aug 26 14:04:25 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2194 (Sun Aug 26 14:04:26 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d681be2e", "local_conf_timestamp": 2194, "host-ts": 2194}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2186 (Sun Aug 26 14:04:17 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2186 (Sun Aug 26 14:04:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "29e75e54", "local_conf_timestamp": 2186, "host-ts": 2186}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:bc151d0d-e429-45ec-b326-c4e57b04f706:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:bc151d0d-e429-45ec-b326-c4e57b04f706:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 8206d8e2 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 8206d8e2 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 8206d8e2 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2204 (Sun Aug 26 14:04:36 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2205 (Sun Aug 26 14:04:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "34bf6954", "local_conf_timestamp": 2205, "host-ts": 2204}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2196 (Sun Aug 26 14:04:27 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2197 (Sun Aug 26 14:04:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "4c9c7607", "local_conf_timestamp": 2197, "host-ts": 2196}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8f21eb76-51e0-4706-a017-4b06b617459f:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:8f21eb76-51e0-4706-a017-4b06b617459f:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 845130a2 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 845130a2 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 845130a2 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2214 (Sun Aug 26 14:04:46 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2215 (Sun Aug 26 14:04:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9152fdda", "local_conf_timestamp": 2215, "host-ts": 2214}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2206 (Sun Aug 26 14:04:37 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2206 (Sun Aug 26 14:04:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5c524054", "local_conf_timestamp": 2206, "host-ts": 2206}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:126ecf00-037c-40ed-84a6-3b75751980dc:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:126ecf00-037c-40ed-84a6-3b75751980dc:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 86951798 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 86951798 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 86951798 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2214 (Sun Aug 26 14:04:46 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2215 (Sun Aug 26 14:04:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9152fdda", "local_conf_timestamp": 2215, "host-ts": 2214}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2206 (Sun Aug 26 14:04:37 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2206 (Sun Aug 26 14:04:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5c524054", "local_conf_timestamp": 2206, "host-ts": 2206}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:e1f2dd2f-6ec9-4f89-af40-88105a9034e7:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:e1f2dd2f-6ec9-4f89-af40-88105a9034e7:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 891db2a4 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 891db2a4 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 891db2a4 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2214 (Sun Aug 26 14:04:46 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2215 (Sun Aug 26 14:04:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9152fdda", "local_conf_timestamp": 2215, "host-ts": 2214}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2206 (Sun Aug 26 14:04:37 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2206 (Sun Aug 26 14:04:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5c524054", "local_conf_timestamp": 2206, "host-ts": 2206}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:4c0118ba-3f9f-4e7a-a34e-c86745110252:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:4c0118ba-3f9f-4e7a-a34e-c86745110252:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 8b66f4bc on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 8b66f4bc on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 8b66f4bc on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2224 (Sun Aug 26 14:04:55 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2224 (Sun Aug 26 14:04:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "06c7c260", "local_conf_timestamp": 2224, "host-ts": 2224}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2216 (Sun Aug 26 14:04:47 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2217 (Sun Aug 26 14:04:48 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "39296807", "local_conf_timestamp": 2217, "host-ts": 2216}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:29c432bf-3e91-4b02-979e-bc4ada4b5e31:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:29c432bf-3e91-4b02-979e-bc4ada4b5e31:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 8db71878 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 8db71878 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 8db71878 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2224 (Sun Aug 26 14:04:55 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2224 (Sun Aug 26 14:04:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "06c7c260", "local_conf_timestamp": 2224, "host-ts": 2224}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2216 (Sun Aug 26 14:04:47 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2217 (Sun Aug 26 14:04:48 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "39296807", "local_conf_timestamp": 2217, "host-ts": 2216}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8ca20943-46ec-468f-866b-926e9c3e7ea6:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:8ca20943-46ec-468f-866b-926e9c3e7ea6:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 90019b26 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 90019b26 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 90019b26 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2234 (Sun Aug 26 14:05:05 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2234 (Sun Aug 26 14:05:05 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "14bbdaa5", "local_conf_timestamp": 2234, "host-ts": 2234}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2226 (Sun Aug 26 14:04:57 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2226 (Sun Aug 26 14:04:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "78aa71de", "local_conf_timestamp": 2226, "host-ts": 2226}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8fca79b1-660e-49e2-83b8-3fc306aa140c:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:8fca79b1-660e-49e2-83b8-3fc306aa140c:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 9269ceba on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9269ceba on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 9269ceba on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2234 (Sun Aug 26 14:05:05 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2234 (Sun Aug 26 14:05:05 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "14bbdaa5", "local_conf_timestamp": 2234, "host-ts": 2234}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2226 (Sun Aug 26 14:04:57 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2226 (Sun Aug 26 14:04:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "78aa71de", "local_conf_timestamp": 2226, "host-ts": 2226}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:758d9896-bd87-4dbf-8748-35e94f75dd00:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:758d9896-bd87-4dbf-8748-35e94f75dd00:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 94aa1d2e on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 94aa1d2e on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 94aa1d2e on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2234 (Sun Aug 26 14:05:05 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2234 (Sun Aug 26 14:05:05 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "14bbdaa5", "local_conf_timestamp": 2234, "host-ts": 2234}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2226 (Sun Aug 26 14:04:57 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2226 (Sun Aug 26 14:04:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "78aa71de", "local_conf_timestamp": 2226, "host-ts": 2226}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:360edee2-2fd9-4382-8331-03ae68226f1b:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:360edee2-2fd9-4382-8331-03ae68226f1b:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 971268be on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 971268be on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 971268be on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2244 (Sun Aug 26 14:05:16 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2245 (Sun Aug 26 14:05:16 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1cc8a068", "local_conf_timestamp": 2245, "host-ts": 2244}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2236 (Sun Aug 26 14:05:07 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2236 (Sun Aug 26 14:05:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6ad6691b", "local_conf_timestamp": 2236, "host-ts": 2236}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:7456e721-7a0a-46da-bbb0-770bbc09ac54:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:7456e721-7a0a-46da-bbb0-770bbc09ac54:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 997f8ad2 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 997f8ad2 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 997f8ad2 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2244 (Sun Aug 26 14:05:16 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2245 (Sun Aug 26 14:05:16 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1cc8a068", "local_conf_timestamp": 2245, "host-ts": 2244}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2236 (Sun Aug 26 14:05:07 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2236 (Sun Aug 26 14:05:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6ad6691b", "local_conf_timestamp": 2236, "host-ts": 2236}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:536fbae0-0bd0-478e-a20e-325a685274fa:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:536fbae0-0bd0-478e-a20e-325a685274fa:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 9be04cf8 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9be04cf8 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 9be04cf8 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2244 (Sun Aug 26 14:05:16 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2245 (Sun Aug 26 14:05:16 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1cc8a068", "local_conf_timestamp": 2245, "host-ts": 2244}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2236 (Sun Aug 26 14:05:07 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2236 (Sun Aug 26 14:05:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6ad6691b", "local_conf_timestamp": 2236, "host-ts": 2236}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:e916b2dc-2d57-4571-83ea-a03b5c863fb9:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:e916b2dc-2d57-4571-83ea-a03b5c863fb9:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 9e63cc5c on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9e63cc5c on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 9e63cc5c on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2254 (Sun Aug 26 14:05:25 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2255 (Sun Aug 26 14:05:26 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1e8c2c84", "local_conf_timestamp": 2255, "host-ts": 2254}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2246 (Sun Aug 26 14:05:17 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2247 (Sun Aug 26 14:05:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "62a513d6", "local_conf_timestamp": 2247, "host-ts": 2246}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:361ba153-6e78-49de-990c-9f8461794e77:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:361ba153-6e78-49de-990c-9f8461794e77:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running a0e98318 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a0e98318 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command a0e98318 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2254 (Sun Aug 26 14:05:25 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2255 (Sun Aug 26 14:05:26 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1e8c2c84", "local_conf_timestamp": 2255, "host-ts": 2254}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2246 (Sun Aug 26 14:05:17 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2247 (Sun Aug 26 14:05:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "62a513d6", "local_conf_timestamp": 2247, "host-ts": 2246}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ad5a7a91-e1ae-47a9-bee1-80971fcc5f22:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:ad5a7a91-e1ae-47a9-bee1-80971fcc5f22:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running a354845e on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a354845e on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command a354845e on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2264 (Sun Aug 26 14:05:36 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2265 (Sun Aug 26 14:05:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "280805cb", "local_conf_timestamp": 2265, "host-ts": 2264}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2256 (Sun Aug 26 14:05:27 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2256 (Sun Aug 26 14:05:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "07de3b85", "local_conf_timestamp": 2256, "host-ts": 2256}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:b5e00fb8-336a-4bc8-82d1-7ab62556304b:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:b5e00fb8-336a-4bc8-82d1-7ab62556304b:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running a5a62c44 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a5a62c44 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command a5a62c44 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2264 (Sun Aug 26 14:05:36 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2265 (Sun Aug 26 14:05:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "280805cb", "local_conf_timestamp": 2265, "host-ts": 2264}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2256 (Sun Aug 26 14:05:27 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2256 (Sun Aug 26 14:05:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "07de3b85", "local_conf_timestamp": 2256, "host-ts": 2256}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:c942e768-03b4-4e0a-b5a1-8f9bee1f1541:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:c942e768-03b4-4e0a-b5a1-8f9bee1f1541:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running a7ff984a on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a7ff984a on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command a7ff984a on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2264 (Sun Aug 26 14:05:36 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2265 (Sun Aug 26 14:05:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "280805cb", "local_conf_timestamp": 2265, "host-ts": 2264}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2256 (Sun Aug 26 14:05:27 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2256 (Sun Aug 26 14:05:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "07de3b85", "local_conf_timestamp": 2256, "host-ts": 2256}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:318bc110-f27d-496e-b518-863d03725adf:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:318bc110-f27d-496e-b518-863d03725adf:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running aa43905c on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command aa43905c on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command aa43905c on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2274 (Sun Aug 26 14:05:46 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2275 (Sun Aug 26 14:05:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "3a741d0e", "local_conf_timestamp": 2275, "host-ts": 2274}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2266 (Sun Aug 26 14:05:37 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2267 (Sun Aug 26 14:05:38 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "en\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f3c24244cf8>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 187, in <lambda>\n for k, v in _get_he_status(host).items()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status\n raise RuntimeError(\'could not parse JSON: %s\' % ret.out)\nRuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2274 (Sun Aug 26 14:05:46 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2275 (Sun Aug 26 14:05:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "3a741d0e", "local_conf_timestamp": 2275, "host-ts": 2274}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2266 (Sun Aug 26 14:05:37 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2267 (Sun Aug 26 14:05:38 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "en\n--------------------- >> end captured logging << ---------------------'
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.2/512/ Build Number: 512 Build Status: Fixed Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #510 [Roy Golan] Introduce openshift-on-ovirt Changes for Build #511 [Roy Golan] Introduce openshift-on-ovirt Changes for Build #512 [Roy Golan] Introduce openshift-on-ovirt ----------------- Failed Tests: ----------------- All tests passed
participants (1)
-
jenkins＠jenkins.phx.ovirt.org