[oVirt Jenkins] ovirt-system-tests_network-suite-master - Build #
741 - Failure!
by jenkins＠jenkins.phx.ovirt.org
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-master/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-master/741/
Build Number: 741
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #741
[Eitan Raviv] networking: sync networks across cluster
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: network-suite-master.tests.ovs.test_ovn_physnet.test_connect_vm_to_external_physnet
Error Message:
test setup failure
Stack Trace:
system = <lib.system.SDKSystemRoot object at 0x7f4ae66508d0>
ovs_cluster = <lib.clusterlib.Cluster object at 0x7f4ad70fff10>
default_cluster = <lib.clusterlib.Cluster object at 0x7f4ae6650b50>
default_data_center = <lib.datacenterlib.DataCenter object at 0x7f4ad7bb0cd0>
@pytest.fixture(scope='session')
def host_in_ovs_cluster(
system, ovs_cluster, default_cluster, default_data_center):
host_id = default_cluster.host_ids()[0]
host = hostlib.Host(system)
host.import_by_id(host_id)
host.wait_for_up_status(timeout=hostlib.HOST_TIMEOUT_LONG)
with hostlib.change_cluster(host, ovs_cluster):
host.sync_all_networks()
default_data_center.wait_for_up_status()
> yield host
network-suite-master/fixtures/host.py:73:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib64/python2.7/contextlib.py:35: in __exit__
self.gen.throw(type, value, traceback)
network-suite-master/lib/hostlib.py:84: in change_cluster
host.change_cluster(original_cluster)
network-suite-master/lib/hostlib.py:141: in change_cluster
self.deactivate()
network-suite-master/lib/hostlib.py:138: in deactivate
self.wait_for_maintenance_status()
network-suite-master/lib/hostlib.py:230: in wait_for_maintenance_status
success_criteria=lambda s: s == HostStatus.MAINTENANCE)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
exec_func = <function <lambda> at 0x7f4ad71b3050>, exec_func_args = ()
success_criteria = <function <lambda> at 0x7f4ad71b3848>
error_criteria = <function <lambda> at 0x7f4aead18b90>, timeout = 120
def sync(exec_func,
exec_func_args,
success_criteria=lambda result: True,
error_criteria=lambda error: True,
timeout=DEFAULT_TIMEOUT):
"""Sync an operation until it either:
- succeeds (according to the success_criteria specified)
- fails due to timing out (after the specified timeout)
- fails due to a terminal error (according to the error_criteria specified)
A caller may specifiy a success_criteria function that should return:
- False if the sync should continue to retry
- True if the sync should terminate immediately
A caller may also specify an error_criteria function if the sync
should continue to retry when the operation fails with an anticipated
exception. This function will be called back with the exception and
should return:
- False if the sync should continue to retry
- True if the sync should stop and the exception raised back to the caller
By default, both success_criteria and error_criteria return True, causing
all results and all errors to return and raise respectively. The default
timeout is 120 seconds.
:param exec_func: callable
:param exec_func_args: tuple/dict
:param success_criteria: callable
:param error_criteria: callable
:param timeout: int
:return: the result of running the exec_func
"""
end_time = _monothonic_time() + timeout
if isinstance(exec_func_args, collections.Mapping):
kwargs = exec_func_args
args = ()
else:
args = exec_func_args
kwargs = {}
try:
result = exec_func(*args, **kwargs)
except Exception as e:
if error_criteria(e):
raise
result = e
else:
if success_criteria(result):
return result
while _monothonic_time() < end_time:
time.sleep(3)
try:
result = exec_func(*args, **kwargs)
except Exception as e:
if error_criteria(e):
raise
result = e
else:
if success_criteria(result):
return result
> raise Timeout(result)
E Timeout: Last evaluated result: up
network-suite-master/lib/syncutil.py:104: Timeout
6 years, 4 months
- 1
- 1
oVirt infra daily report - unstable production jobs - 638
by jenkins＠jenkins.phx.ovirt.org
Good morning!
Attached is the HTML page with the jenkins status report. You can see it also here:
- http://jenkins.ovirt.org/job/system_jenkins-report/638//artifact/exported...
Cheers,
Jenkins
6 years, 4 months
- 1
- 0
[CQ]: 95282, 3 (ovirt-engine) failed "ovirt-master" system tests,
but isn't the failure root cause
by oVirt Jenkins
A system test invoked by the "ovirt-master" change queue including change
95282,3 (ovirt-engine) failed. However, this change seems not to be the root
cause for this failure. Change 95190,2 (ovirt-engine) that this change depends
on or is based on, was detected as the cause of the testing failures.
This change had been removed from the testing queue. Artifacts built from this
change will not be released until either change 95190,2 (ovirt-engine) is fixed
and this change is updated to refer to or rebased on the fixed version, or this
change is modified to no longer depend on it.
For further details about the change see:
https://gerrit.ovirt.org/#/c/95282/3
For further details about the change that seems to be the root cause behind the
testing failures see:
https://gerrit.ovirt.org/#/c/95190/2
For failed test results see:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11078/
6 years, 4 months
- 1
- 0
[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-4.2 - Build #
475 - Failure!
by jenkins＠jenkins.phx.ovirt.org
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/475/
Build Number: 475
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #475
[Roy Golan] Add README to openshift-on-ovirt suite
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: 012_local_maintenance_sdk.local_maintenance
Error Message:
143
-------------------- >> begin captured logging << --------------------
root: INFO: * Waiting For System Stability...
root: INFO: * Performing Deactivation...
root: INFO: * Performing Activation...
root: INFO: * Waiting For System Stability...
cli: DEBUG: signal 15 was caught
--------------------- >> end captured logging << ---------------------
Stack Trace:
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper
prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/012_local_maintenance_sdk.py", line 79, in local_maintenance
time.sleep(wait_value)
File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler
sys.exit(128 + signum)
'143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nroot: INFO: * Performing Deactivation...\nroot: INFO: * Performing Activation...\nroot: INFO: * Waiting For System Stability...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'
6 years, 4 months
- 1
- 1
[oVirt Jenkins] ovirt-system-tests_he-basic-ansible-suite-4.2 -
Build # 730 - Failure!
by jenkins＠jenkins.phx.ovirt.org
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4.2/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4....
Build Number: 730
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #730
[Roy Golan] Add README to openshift-on-ovirt suite
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: 008_restart_he_vm.restart_he_vm
Error Message:
could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2675 (Thu Nov 8 03:19:13 2018)\nhost-id=1\nscore=3111\nvm_conf_refresh_time=2675 (Thu Nov 8 03:19:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "be3d40a1", "local_conf_timestamp": 2675, "host-ts": 2675}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2673 (Thu Nov 8 03:19:12 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2673 (Thu Nov 8 03:19:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2
-------------------- >> begin captured logging << --------------------
lago.ssh: DEBUG: start task:053eb136-48b1-45d4-a8b7-1b6f74536f40:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:053eb136-48b1-45d4-a8b7-1b6f74536f40:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running b6cf4628 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command b6cf4628 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command b6cf4628 on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2555 (Thu Nov 8 03:17:13 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2555 (Thu Nov 8 03:17:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "91a2a003", "local_conf_timestamp": 2555, "host-ts": 2555}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2553 (Thu Nov 8 03:17:12 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2553 (Thu Nov 8 03:17:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a7e651e6", "local_conf_timestamp": 2553, "host-ts": 2553}, "global_maintenance": true}
root: INFO: * Shutting down HE VM on host: lago-he-basic-ansible-suite-4-2-host-0
lago.ssh: DEBUG: start task:48e52f8d-78cd-4c3e-9bdb-f862ac50a1e5:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:48e52f8d-78cd-4c3e-9bdb-f862ac50a1e5:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running b7683c5c on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-shutdown
lago.ssh: DEBUG: Command b7683c5c on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
root: INFO: * Command succeeded
root: INFO: * Waiting for VM to be down...
lago.ssh: DEBUG: start task:b8b1cea3-0cce-43d1-8872-84dc6282c96d:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:b8b1cea3-0cce-43d1-8872-84dc6282c96d:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running b83d8ef2 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command b83d8ef2 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command b83d8ef2 on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2555 (Thu Nov 8 03:17:13 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2555 (Thu Nov 8 03:17:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "91a2a003", "local_conf_timestamp": 2555, "host-ts": 2555}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2553 (Thu Nov 8 03:17:12 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2553 (Thu Nov 8 03:17:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a7e651e6", "local_conf_timestamp": 2553, "host-ts": 2553}, "global_maintenance": true}
lago.ssh: DEBUG: start task:e6d043c5-156b-4288-a7a3-f24161654e22:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:e6d043c5-156b-4288-a7a3-f24161654e22:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running bed30274 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command bed30274 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command bed30274 on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2575 (Thu Nov 8 03:17:33 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2575 (Thu Nov 8 03:17:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "73a2b866", "local_conf_timestamp": 2575, "host-ts": 2575}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2573 (Thu Nov 8 03:17:31 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2573 (Thu Nov 8 03:17:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "904ac61d", "local_conf_timestamp": 2573, "host-ts": 2573}, "global_maintenance": true}
root: INFO: * VM is down.
root: INFO: * Stopping services...
lago.ssh: DEBUG: start task:4d492e84-b98f-410a-9186-9da4ad96953b:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:4d492e84-b98f-410a-9186-9da4ad96953b:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running bf583408 on lago-he-basic-ansible-suite-4-2-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent
lago.ssh: DEBUG: Command bf583408 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
root: INFO: * Starting services...
lago.ssh: DEBUG: start task:570131db-44f3-4f33-9b1c-1ca470f7db23:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:570131db-44f3-4f33-9b1c-1ca470f7db23:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running c7ea8f58 on lago-he-basic-ansible-suite-4-2-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent
lago.ssh: DEBUG: Command c7ea8f58 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
root: INFO: * Waiting for agent to be ready...
lago.ssh: DEBUG: start task:15866098-59b0-4b3e-aca7-9f87b0faaa09:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:15866098-59b0-4b3e-aca7-9f87b0faaa09:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running ca1f0a7e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command ca1f0a7e on lago-he-basic-ansible-suite-4-2-host-0 returned with 1
lago.ssh: DEBUG: Command ca1f0a7e on lago-he-basic-ansible-suite-4-2-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:9d69a85b-cf95-4cd5-acf7-849b3eef94ae:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:9d69a85b-cf95-4cd5-acf7-849b3eef94ae:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running d06eaa7e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command d06eaa7e on lago-he-basic-ansible-suite-4-2-host-0 returned with 1
lago.ssh: DEBUG: Command d06eaa7e on lago-he-basic-ansible-suite-4-2-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:f122fda8-a80e-420d-8427-bd667eaf09c3:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:f122fda8-a80e-420d-8427-bd667eaf09c3:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running d2bcad8a on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command d2bcad8a on lago-he-basic-ansible-suite-4-2-host-0 returned with 1
lago.ssh: DEBUG: Command d2bcad8a on lago-he-basic-ansible-suite-4-2-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:4f32a250-ab55-46c5-b228-a5ec3477a39c:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:4f32a250-ab55-46c5-b228-a5ec3477a39c:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running d4e22d24 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command d4e22d24 on lago-he-basic-ansible-suite-4-2-host-0 returned with 1
lago.ssh: DEBUG: Command d4e22d24 on lago-he-basic-ansible-suite-4-2-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:fa589610-73ae-4944-9b08-cfcc0c7a3256:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:fa589610-73ae-4944-9b08-cfcc0c7a3256:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running d708a61e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command d708a61e on lago-he-basic-ansible-suite-4-2-host-0 returned with 1
lago.ssh: DEBUG: Command d708a61e on lago-he-basic-ansible-suite-4-2-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:31625e1e-23bb-43b9-9ab7-910a7dafc5ed:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:31625e1e-23bb-43b9-9ab7-910a7dafc5ed:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running d93645ae on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command d93645ae on lago-he-basic-ansible-suite-4-2-host-0 returned with 1
lago.ssh: DEBUG: Command d93645ae on lago-he-basic-ansible-suite-4-2-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:6077407c-ae81-4203-bfe4-007bae8817fb:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:6077407c-ae81-4203-bfe4-007bae8817fb:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running db5f2030 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command db5f2030 on lago-he-basic-ansible-suite-4-2-host-0 returned with 1
lago.ssh: DEBUG: Command db5f2030 on lago-he-basic-ansible-suite-4-2-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:77846988-89e4-45f0-8aba-cacb985bf1b9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:77846988-89e4-45f0-8aba-cacb985bf1b9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running dd889382 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command dd889382 on lago-he-basic-ansible-suite-4-2-host-0 returned with 1
lago.ssh: DEBUG: Command dd889382 on lago-he-basic-ansible-suite-4-2-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:f1925127-d380-43cc-86ea-4f90bcb5b63d:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:f1925127-d380-43cc-86ea-4f90bcb5b63d:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running dfb22bbe on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command dfb22bbe on lago-he-basic-ansible-suite-4-2-host-0 returned with 1
lago.ssh: DEBUG: Command dfb22bbe on lago-he-basic-ansible-suite-4-2-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:4edc99b2-4a65-4147-b24d-56bbc8046b66:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:4edc99b2-4a65-4147-b24d-56bbc8046b66:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running e1d9e5e4 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command e1d9e5e4 on lago-he-basic-ansible-suite-4-2-host-0 returned with 1
lago.ssh: DEBUG: Command e1d9e5e4 on lago-he-basic-ansible-suite-4-2-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:70bfaba1-ad03-4f96-918f-1e2793ee7427:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:70bfaba1-ad03-4f96-918f-1e2793ee7427:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running e40a02a4 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command e40a02a4 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command e40a02a4 on lago-he-basic-ansible-suite-4-2-host-0 output:
!! Cluster is in GLOBAL MAINTENANCE mode !!
--== Host 1 status ==--
conf_on_shared_storage : True
Status up-to-date : True
Hostname : lago-he-basic-ansible-suite-4-2-host-0
Host ID : 1
Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}
Score : 0
stopped : False
Local maintenance : False
crc32 : 36bd3d62
local_conf_timestamp : 2635
Host timestamp : 2635
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=2635 (Thu Nov 8 03:18:33 2018)
host-id=1
score=0
vm_conf_refresh_time=2635 (Thu Nov 8 03:18:33 2018)
conf_on_shared_storage=True
maintenance=False
state=ReinitializeFSM
stopped=False
--== Host 2 status ==--
conf_on_shared_storage : True
Status up-to-date : False
Hostname : lago-he-basic-ansible-suite-4-2-host-1
Host ID : 2
Engine status : unknown stale-data
Score : 3400
stopped : False
Local maintenance : False
crc32 : 2ddfae49
local_conf_timestamp : 2633
Host timestamp : 2633
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=2633 (Thu Nov 8 03:18:32 2018)
host-id=2
score=3400
vm_conf_refresh_time=2633 (Thu Nov 8 03:18:32 2018)
conf_on_shared_storage=True
maintenance=False
state=GlobalMaintenance
stopped=False
!! Cluster is in GLOBAL MAINTENANCE mode !!
root: INFO: * Agent is ready.
root: INFO: * Starting VM...
lago.ssh: DEBUG: start task:ff98dc0f-e7fc-4cb8-b035-7af590da59dd:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:ff98dc0f-e7fc-4cb8-b035-7af590da59dd:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running e4911c3a on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-start
lago.ssh: DEBUG: Command e4911c3a on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command e4911c3a on lago-he-basic-ansible-suite-4-2-host-0 output:
VM exists and is down, cleaning up and restarting
root: INFO: * Command succeeded
root: INFO: * Waiting for VM to be UP...
lago.ssh: DEBUG: start task:acf835ef-f2f1-4150-9e4a-69f3b18687fa:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:acf835ef-f2f1-4150-9e4a-69f3b18687fa:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running e5c3aa46 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command e5c3aa46 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command e5c3aa46 on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2635 (Thu Nov 8 03:18:33 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=2635 (Thu Nov 8 03:18:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=ReinitializeFSM\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "36bd3d62", "local_conf_timestamp": 2635, "host-ts": 2635}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2633 (Thu Nov 8 03:18:32 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2633 (Thu Nov 8 03:18:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2ddfae49", "local_conf_timestamp": 2633, "host-ts": 2633}, "global_maintenance": true}
lago.ssh: DEBUG: start task:eb7dba65-0edb-4c63-9c01-22e619db3e60:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:eb7dba65-0edb-4c63-9c01-22e619db3e60:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running ec372e3e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command ec372e3e on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command ec372e3e on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2645 (Thu Nov 8 03:18:44 2018)\nhost-id=1\nscore=3111\nvm_conf_refresh_time=2645 (Thu Nov 8 03:18:44 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "0df3ca26", "local_conf_timestamp": 2645, "host-ts": 2645}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2643 (Thu Nov 8 03:18:42 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2643 (Thu Nov 8 03:18:42 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8344de99", "local_conf_timestamp": 2643, "host-ts": 2643}, "global_maintenance": true}
root: INFO: * VM is UP.
root: INFO: * Waiting for engine to start...
lago.ssh: DEBUG: start task:8bf6658f-e1f4-4aa8-a773-b2d92f60e02a:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:8bf6658f-e1f4-4aa8-a773-b2d92f60e02a:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running ecaca8ee on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command ecaca8ee on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command ecaca8ee on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2645 (Thu Nov 8 03:18:44 2018)\nhost-id=1\nscore=3111\nvm_conf_refresh_time=2645 (Thu Nov 8 03:18:44 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "0df3ca26", "local_conf_timestamp": 2645, "host-ts": 2645}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2643 (Thu Nov 8 03:18:42 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2643 (Thu Nov 8 03:18:42 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8344de99", "local_conf_timestamp": 2643, "host-ts": 2643}, "global_maintenance": true}
lago.ssh: DEBUG: start task:71c5108d-271c-4aad-8aec-0a8b7337f359:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:71c5108d-271c-4aad-8aec-0a8b7337f359:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running f330d5aa on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command f330d5aa on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command f330d5aa on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2655 (Thu Nov 8 03:18:54 2018)\nhost-id=1\nscore=3111\nvm_conf_refresh_time=2656 (Thu Nov 8 03:18:54 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "4c901fde", "local_conf_timestamp": 2656, "host-ts": 2655}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2653 (Thu Nov 8 03:18:51 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2653 (Thu Nov 8 03:18:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "752a1644", "local_conf_timestamp": 2653, "host-ts": 2653}, "global_maintenance": true}
lago.ssh: DEBUG: start task:e68357ea-ea52-44f4-b67e-e22f47d6a2ac:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:e68357ea-ea52-44f4-b67e-e22f47d6a2ac:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running f5a2235c on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command f5a2235c on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command f5a2235c on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2665 (Thu Nov 8 03:19:03 2018)\nhost-id=1\nscore=3111\nvm_conf_refresh_time=2665 (Thu Nov 8 03:19:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "6657c4e3", "local_conf_timestamp": 2665, "host-ts": 2665}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2663 (Thu Nov 8 03:19:02 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2663 (Thu Nov 8 03:19:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b4e84962", "local_conf_timestamp": 2663, "host-ts": 2663}, "global_maintenance": true}
lago.ssh: DEBUG: start task:436dfa11-97d6-4095-804e-566329a5fdee:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:436dfa11-97d6-4095-804e-566329a5fdee:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running f7ff723a on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command f7ff723a on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command f7ff723a on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2665 (Thu Nov 8 03:19:03 2018)\nhost-id=1\nscore=3111\nvm_conf_refresh_time=2665 (Thu Nov 8 03:19:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "6657c4e3", "local_conf_timestamp": 2665, "host-ts": 2665}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2663 (Thu Nov 8 03:19:02 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2663 (Thu Nov 8 03:19:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b4e84962", "local_conf_timestamp": 2663, "host-ts": 2663}, "global_maintenance": true}
lago.ssh: DEBUG: start task:de0e96fe-575e-4a46-82f8-4e3d5a85ca1f:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:de0e96fe-575e-4a46-82f8-4e3d5a85ca1f:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running fa442a36 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command fa442a36 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command fa442a36 on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2675 (Thu Nov 8 03:19:13 2018)\nhost-id=1\nscore=3111\nvm_conf_refresh_time=2675 (Thu Nov 8 03:19:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "be3d40a1", "local_conf_timestamp": 2675, "host-ts": 2675}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2673 (Thu Nov 8 03:19:12 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2673 (Thu Nov 8 03:19:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "428681bf", "local_conf_timestamp": 2673, "host-ts": 2673}, "global_maintenance": true}
lago.ssh: DEBUG: start task:97f350dc-f503-4726-84cb-61ad3b8166bc:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:97f350dc-f503-4726-84cb-61ad3b8166bc:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running fc9a48b0 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command fc9a48b0 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command fc9a48b0 on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2675 (Thu Nov 8 03:19:13 2018)\nhost-id=1\nscore=3111\nvm_conf_refresh_time=2675 (Thu Nov 8 03:19:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "be3d40a1", "local_conf_timestamp": 2675, "host-ts": 2675}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2673 (Thu Nov 8 03:19:12 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2673 (Thu Nov 8 03:19:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2
ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f1389d43e60>
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within
res = func()
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/008_restart_he_vm.py", line 187, in <lambda>
for k, v in _get_he_status(host).items()
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status
raise RuntimeError('could not parse JSON: %s' % ret.out)
RuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2675 (Thu Nov 8 03:19:13 2018)\nhost-id=1\nscore=3111\nvm_conf_refresh_time=2675 (Thu Nov 8 03:19:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "be3d40a1", "local_conf_timestamp": 2675, "host-ts": 2675}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2673 (Thu Nov 8 03:19:12 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2673 (Thu Nov 8 03:19:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2
--------------------- >> end captured logging << ---------------------
Stack Trace:
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/008_restart_he_vm.py", line 53, in restart_he_vm
_wait_for_engine_health(host)
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/008_restart_he_vm.py", line 185, in _wait_for_engine_health
testlib.assert_true_within_long(lambda: any(
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long
assert_equals_within_long(func, True, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long
func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within
res = func()
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/008_restart_he_vm.py", line 187, in <lambda>
for k, v in _get_he_status(host).items()
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status
raise RuntimeError('could not parse JSON: %s' % ret.out)
'could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2675 (Thu Nov 8 03:19:13 2018)\\nhost-id=1\\nscore=3111\\nvm_conf_refresh_time=2675 (Thu Nov 8 03:19:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "be3d40a1", "local_conf_timestamp": 2675, "host-ts": 2675}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2673 (Thu Nov 8 03:19:12 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2673 (Thu Nov 8 03:19:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:053eb136-48b1-45d4-a8b7-1b6f74536f40:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:053eb136-48b1-45d4-a8b7-1b6f74536f40:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b6cf4628 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b6cf4628 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command b6cf4628 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2555 (Thu Nov 8 03:17:13 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2555 (Thu Nov 8 03:17:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "91a2a003", "local_conf_timestamp": 2555, "host-ts": 2555}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2553 (Thu Nov 8 03:17:12 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2553 (Thu Nov 8 03:17:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a7e651e6", "local_conf_timestamp": 2553, "host-ts": 2553}, "global_maintenance": true}\n\nroot: INFO: * Shutting down HE VM on host: lago-he-basic-ansible-suite-4-2-host-0\nlago.ssh: DEBUG: start task:48e52f8d-78cd-4c3e-9bdb-f862ac50a1e5:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:48e52f8d-78cd-4c3e-9bdb-f862ac50a1e5:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b7683c5c on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-shutdown\nlago.ssh: DEBUG: Command b7683c5c on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be down...\nlago.ssh: DEBUG: start task:b8b1cea3-0cce-43d1-8872-84dc6282c96d:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:b8b1cea3-0cce-43d1-8872-84dc6282c96d:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b83d8ef2 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b83d8ef2 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command b83d8ef2 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2555 (Thu Nov 8 03:17:13 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2555 (Thu Nov 8 03:17:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "91a2a003", "local_conf_timestamp": 2555, "host-ts": 2555}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2553 (Thu Nov 8 03:17:12 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2553 (Thu Nov 8 03:17:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a7e651e6", "local_conf_timestamp": 2553, "host-ts": 2553}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:e6d043c5-156b-4288-a7a3-f24161654e22:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:e6d043c5-156b-4288-a7a3-f24161654e22:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running bed30274 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command bed30274 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command bed30274 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2575 (Thu Nov 8 03:17:33 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2575 (Thu Nov 8 03:17:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "73a2b866", "local_conf_timestamp": 2575, "host-ts": 2575}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2573 (Thu Nov 8 03:17:31 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2573 (Thu Nov 8 03:17:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "904ac61d", "local_conf_timestamp": 2573, "host-ts": 2573}, "global_maintenance": true}\n\nroot: INFO: * VM is down.\nroot: INFO: * Stopping services...\nlago.ssh: DEBUG: start task:4d492e84-b98f-410a-9186-9da4ad96953b:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:4d492e84-b98f-410a-9186-9da4ad96953b:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running bf583408 on lago-he-basic-ansible-suite-4-2-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command bf583408 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nroot: INFO: * Starting services...\nlago.ssh: DEBUG: start task:570131db-44f3-4f33-9b1c-1ca470f7db23:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:570131db-44f3-4f33-9b1c-1ca470f7db23:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running c7ea8f58 on lago-he-basic-ansible-suite-4-2-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command c7ea8f58 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nroot: INFO: * Waiting for agent to be ready...\nlago.ssh: DEBUG: start task:15866098-59b0-4b3e-aca7-9f87b0faaa09:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:15866098-59b0-4b3e-aca7-9f87b0faaa09:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running ca1f0a7e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command ca1f0a7e on lago-he-basic-ansible-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command ca1f0a7e on lago-he-basic-ansible-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:9d69a85b-cf95-4cd5-acf7-849b3eef94ae:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:9d69a85b-cf95-4cd5-acf7-849b3eef94ae:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running d06eaa7e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command d06eaa7e on lago-he-basic-ansible-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command d06eaa7e on lago-he-basic-ansible-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:f122fda8-a80e-420d-8427-bd667eaf09c3:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:f122fda8-a80e-420d-8427-bd667eaf09c3:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running d2bcad8a on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command d2bcad8a on lago-he-basic-ansible-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command d2bcad8a on lago-he-basic-ansible-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:4f32a250-ab55-46c5-b228-a5ec3477a39c:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:4f32a250-ab55-46c5-b228-a5ec3477a39c:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running d4e22d24 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command d4e22d24 on lago-he-basic-ansible-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command d4e22d24 on lago-he-basic-ansible-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:fa589610-73ae-4944-9b08-cfcc0c7a3256:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:fa589610-73ae-4944-9b08-cfcc0c7a3256:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running d708a61e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command d708a61e on lago-he-basic-ansible-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command d708a61e on lago-he-basic-ansible-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:31625e1e-23bb-43b9-9ab7-910a7dafc5ed:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:31625e1e-23bb-43b9-9ab7-910a7dafc5ed:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running d93645ae on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command d93645ae on lago-he-basic-ansible-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command d93645ae on lago-he-basic-ansible-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:6077407c-ae81-4203-bfe4-007bae8817fb:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:6077407c-ae81-4203-bfe4-007bae8817fb:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running db5f2030 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command db5f2030 on lago-he-basic-ansible-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command db5f2030 on lago-he-basic-ansible-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:77846988-89e4-45f0-8aba-cacb985bf1b9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:77846988-89e4-45f0-8aba-cacb985bf1b9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running dd889382 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command dd889382 on lago-he-basic-ansible-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command dd889382 on lago-he-basic-ansible-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:f1925127-d380-43cc-86ea-4f90bcb5b63d:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:f1925127-d380-43cc-86ea-4f90bcb5b63d:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running dfb22bbe on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command dfb22bbe on lago-he-basic-ansible-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command dfb22bbe on lago-he-basic-ansible-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:4edc99b2-4a65-4147-b24d-56bbc8046b66:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:4edc99b2-4a65-4147-b24d-56bbc8046b66:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running e1d9e5e4 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command e1d9e5e4 on lago-he-basic-ansible-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command e1d9e5e4 on lago-he-basic-ansible-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:70bfaba1-ad03-4f96-918f-1e2793ee7427:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:70bfaba1-ad03-4f96-918f-1e2793ee7427:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running e40a02a4 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command e40a02a4 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command e40a02a4 on lago-he-basic-ansible-suite-4-2-host-0 output:\n \n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\n\n--== Host 1 status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : True\nHostname : lago-he-basic-ansible-suite-4-2-host-0\nHost ID : 1\nEngine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}\nScore : 0\nstopped : False\nLocal maintenance : False\ncrc32 : 36bd3d62\nlocal_conf_timestamp : 2635\nHost timestamp : 2635\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2635 (Thu Nov 8 03:18:33 2018)\n\thost-id=1\n\tscore=0\n\tvm_conf_refresh_time=2635 (Thu Nov 8 03:18:33 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=ReinitializeFSM\n\tstopped=False\n\n\n--== Host 2 status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : False\nHostname : lago-he-basic-ansible-suite-4-2-host-1\nHost ID : 2\nEngine status : unknown stale-data\nScore : 3400\nstopped : False\nLocal maintenance : False\ncrc32 : 2ddfae49\nlocal_conf_timestamp : 2633\nHost timestamp : 2633\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2633 (Thu Nov 8 03:18:32 2018)\n\thost-id=2\n\tscore=3400\n\tvm_conf_refresh_time=2633 (Thu Nov 8 03:18:32 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=GlobalMaintenance\n\tstopped=False\n\n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\nroot: INFO: * Agent is ready.\nroot: INFO: * Starting VM...\nlago.ssh: DEBUG: start task:ff98dc0f-e7fc-4cb8-b035-7af590da59dd:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:ff98dc0f-e7fc-4cb8-b035-7af590da59dd:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running e4911c3a on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-start\nlago.ssh: DEBUG: Command e4911c3a on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command e4911c3a on lago-he-basic-ansible-suite-4-2-host-0 output:\n VM exists and is down, cleaning up and restarting\n\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be UP...\nlago.ssh: DEBUG: start task:acf835ef-f2f1-4150-9e4a-69f3b18687fa:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:acf835ef-f2f1-4150-9e4a-69f3b18687fa:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running e5c3aa46 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e5c3aa46 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command e5c3aa46 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2635 (Thu Nov 8 03:18:33 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=2635 (Thu Nov 8 03:18:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=ReinitializeFSM\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "36bd3d62", "local_conf_timestamp": 2635, "host-ts": 2635}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2633 (Thu Nov 8 03:18:32 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2633 (Thu Nov 8 03:18:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2ddfae49", "local_conf_timestamp": 2633, "host-ts": 2633}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:eb7dba65-0edb-4c63-9c01-22e619db3e60:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:eb7dba65-0edb-4c63-9c01-22e619db3e60:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running ec372e3e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ec372e3e on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command ec372e3e on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2645 (Thu Nov 8 03:18:44 2018)\\nhost-id=1\\nscore=3111\\nvm_conf_refresh_time=2645 (Thu Nov 8 03:18:44 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "0df3ca26", "local_conf_timestamp": 2645, "host-ts": 2645}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2643 (Thu Nov 8 03:18:42 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2643 (Thu Nov 8 03:18:42 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8344de99", "local_conf_timestamp": 2643, "host-ts": 2643}, "global_maintenance": true}\n\nroot: INFO: * VM is UP.\nroot: INFO: * Waiting for engine to start...\nlago.ssh: DEBUG: start task:8bf6658f-e1f4-4aa8-a773-b2d92f60e02a:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:8bf6658f-e1f4-4aa8-a773-b2d92f60e02a:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running ecaca8ee on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ecaca8ee on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command ecaca8ee on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2645 (Thu Nov 8 03:18:44 2018)\\nhost-id=1\\nscore=3111\\nvm_conf_refresh_time=2645 (Thu Nov 8 03:18:44 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "0df3ca26", "local_conf_timestamp": 2645, "host-ts": 2645}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2643 (Thu Nov 8 03:18:42 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2643 (Thu Nov 8 03:18:42 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8344de99", "local_conf_timestamp": 2643, "host-ts": 2643}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:71c5108d-271c-4aad-8aec-0a8b7337f359:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:71c5108d-271c-4aad-8aec-0a8b7337f359:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running f330d5aa on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f330d5aa on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command f330d5aa on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2655 (Thu Nov 8 03:18:54 2018)\\nhost-id=1\\nscore=3111\\nvm_conf_refresh_time=2656 (Thu Nov 8 03:18:54 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "4c901fde", "local_conf_timestamp": 2656, "host-ts": 2655}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2653 (Thu Nov 8 03:18:51 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2653 (Thu Nov 8 03:18:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "752a1644", "local_conf_timestamp": 2653, "host-ts": 2653}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:e68357ea-ea52-44f4-b67e-e22f47d6a2ac:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:e68357ea-ea52-44f4-b67e-e22f47d6a2ac:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running f5a2235c on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f5a2235c on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command f5a2235c on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2665 (Thu Nov 8 03:19:03 2018)\\nhost-id=1\\nscore=3111\\nvm_conf_refresh_time=2665 (Thu Nov 8 03:19:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "6657c4e3", "local_conf_timestamp": 2665, "host-ts": 2665}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2663 (Thu Nov 8 03:19:02 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2663 (Thu Nov 8 03:19:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b4e84962", "local_conf_timestamp": 2663, "host-ts": 2663}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:436dfa11-97d6-4095-804e-566329a5fdee:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:436dfa11-97d6-4095-804e-566329a5fdee:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running f7ff723a on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f7ff723a on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command f7ff723a on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2665 (Thu Nov 8 03:19:03 2018)\\nhost-id=1\\nscore=3111\\nvm_conf_refresh_time=2665 (Thu Nov 8 03:19:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "6657c4e3", "local_conf_timestamp": 2665, "host-ts": 2665}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2663 (Thu Nov 8 03:19:02 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2663 (Thu Nov 8 03:19:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b4e84962", "local_conf_timestamp": 2663, "host-ts": 2663}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:de0e96fe-575e-4a46-82f8-4e3d5a85ca1f:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:de0e96fe-575e-4a46-82f8-4e3d5a85ca1f:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running fa442a36 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fa442a36 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command fa442a36 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2675 (Thu Nov 8 03:19:13 2018)\\nhost-id=1\\nscore=3111\\nvm_conf_refresh_time=2675 (Thu Nov 8 03:19:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "be3d40a1", "local_conf_timestamp": 2675, "host-ts": 2675}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2673 (Thu Nov 8 03:19:12 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2673 (Thu Nov 8 03:19:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "428681bf", "local_conf_timestamp": 2673, "host-ts": 2673}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:97f350dc-f503-4726-84cb-61ad3b8166bc:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:97f350dc-f503-4726-84cb-61ad3b8166bc:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running fc9a48b0 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fc9a48b0 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command fc9a48b0 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2675 (Thu Nov 8 03:19:13 2018)\\nhost-id=1\\nscore=3111\\nvm_conf_refresh_time=2675 (Thu Nov 8 03:19:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "be3d40a1", "local_conf_timestamp": 2675, "host-ts": 2675}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2673 (Thu Nov 8 03:19:12 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2673 (Thu Nov 8 03:19:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f1389d43e60>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/008_restart_he_vm.py", line 187, in <lambda>\n for k, v in _get_he_status(host).items()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status\n raise RuntimeError(\'could not parse JSON: %s\' % ret.out)\nRuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2675 (Thu Nov 8 03:19:13 2018)\\nhost-id=1\\nscore=3111\\nvm_conf_refresh_time=2675 (Thu Nov 8 03:19:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3111, "stopped": false, "maintenance": false, "crc32": "be3d40a1", "local_conf_timestamp": 2675, "host-ts": 2675}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2673 (Thu Nov 8 03:19:12 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2673 (Thu Nov 8 03:19:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2\n--------------------- >> end captured logging << ---------------------'
6 years, 4 months
- 1
- 1
[CQ]: 95290, 4 (ovirt-engine) failed "ovirt-master" system tests,
but isn't the failure root cause
by oVirt Jenkins
A system test invoked by the "ovirt-master" change queue including change
95290,4 (ovirt-engine) failed. However, this change seems not to be the root
cause for this failure. Change 95190,2 (ovirt-engine) that this change depends
on or is based on, was detected as the cause of the testing failures.
This change had been removed from the testing queue. Artifacts built from this
change will not be released until either change 95190,2 (ovirt-engine) is fixed
and this change is updated to refer to or rebased on the fixed version, or this
change is modified to no longer depend on it.
For further details about the change see:
https://gerrit.ovirt.org/#/c/95290/4
For further details about the change that seems to be the root cause behind the
testing failures see:
https://gerrit.ovirt.org/#/c/95190/2
For failed test results see:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11072/
6 years, 4 months
- 1
- 0
[CQ]: 95163, 8 (ovirt-engine) failed "ovirt-master" system tests,
but isn't the failure root cause
by oVirt Jenkins
A system test invoked by the "ovirt-master" change queue including change
95163,8 (ovirt-engine) failed. However, this change seems not to be the root
cause for this failure. Change 95190,2 (ovirt-engine) that this change depends
on or is based on, was detected as the cause of the testing failures.
This change had been removed from the testing queue. Artifacts built from this
change will not be released until either change 95190,2 (ovirt-engine) is fixed
and this change is updated to refer to or rebased on the fixed version, or this
change is modified to no longer depend on it.
For further details about the change see:
https://gerrit.ovirt.org/#/c/95163/8
For further details about the change that seems to be the root cause behind the
testing failures see:
https://gerrit.ovirt.org/#/c/95190/2
For failed test results see:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11066/
6 years, 4 months
- 2
- 1
[JIRA] (OVIRT-2571) master ovirt-engine fails on basic suite, test
remove_vm_pool
by Dafna Ron (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2571?page=com.atlassian.jir... ]
Dafna Ron reassigned OVIRT-2571:
--------------------------------
Assignee: Dominik Holler (was: infra)
> master ovirt-engine fails on basic suite, test remove_vm_pool
> -------------------------------------------------------------
>
> Key: OVIRT-2571
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2571
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Reporter: Dafna Ron
> Assignee: Dominik Holler
> Labels: ost_failures
>
> This actually seems to me to be related to the test and not the patch.
> We try to remove pool while vm is being updated.
> seems like this test might be a network test they should look into this issue.
> Will ask Dominic to look at this first and rule out his patch as the cause.
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11066/
> Patch that failed: https://gerrit.ovirt.org/#/c/95163/
> {noformat}
> 2018-11-08 04:26:42,081-05 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-1) [] Operation Failed: [Cannot remove VM-Pool. VM is being updated.]
> {noformat}
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100095)
6 years, 4 months
- 1
- 0
[JIRA] (OVIRT-2571) master ovirt-engine fails on basic suite, test
remove_vm_pool
by Dafna Ron (oVirt JIRA)
Dafna Ron created OVIRT-2571:
--------------------------------
Summary: master ovirt-engine fails on basic suite, test remove_vm_pool
Key: OVIRT-2571
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2571
Project: oVirt - virtualization made easy
Issue Type: Bug
Reporter: Dafna Ron
Assignee: infra
This actually seems to me to be related to the test and not the patch.
We try to remove pool while vm is being updated.
seems like this test might be a network test they should look into this issue.
Will ask Dominic to look at this first and rule out his patch as the cause.
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11066/
Patch that failed: https://gerrit.ovirt.org/#/c/95163/
{noformat}
2018-11-08 04:26:42,081-05 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-1) [] Operation Failed: [Cannot remove VM-Pool. VM is being updated.]
{noformat}
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100095)
6 years, 4 months
- 1
- 0
[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-master - Build
# 540 - Failure!
by jenkins＠jenkins.phx.ovirt.org
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/540/
Build Number: 540
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #540
[Roy Golan] Add README to openshift-on-ovirt suite
-----------------
Failed Tests:
-----------------
No tests ran.
6 years, 4 months
- 1
- 1