[oVirt Jenkins] ovirt-system-tests_he-basic-ansible-suite-4.2 -
Build # 565 - Failure!
by jenkins@jenkins.phx.ovirt.org
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4.2/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4....
Build Number: 565
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #565
[Milan Zamazal] Update test-vm.ovf file
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: 010_local_maintenance_cli.local_maintenance
Error Message:
could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason
-------------------- >> begin captured logging << --------------------
root: INFO: * Waiting For System Stability...
lago.ssh: DEBUG: start task:0874be8a-4d1f-4242-8c58-343d3bf8718a:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:0874be8a-4d1f-4242-8c58-343d3bf8718a:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running 2d38dc7e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 2d38dc7e on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 2d38dc7e on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3333 (Fri Sep 14 04:21:19 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=3333 (Fri Sep 14 04:21:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b74983cd", "local_conf_timestamp": 3333, "host-ts": 3333}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3331 (Fri Sep 14 04:21:18 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3331 (Fri Sep 14 04:21:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "547f6cbb", "local_conf_timestamp": 3331, "host-ts": 3331}, "global_maintenance": false}
lago.ssh: DEBUG: start task:de4315ba-d73d-4a51-8315-8f960c2a0e35:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:de4315ba-d73d-4a51-8315-8f960c2a0e35:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running 2db65230 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local
lago.ssh: DEBUG: Command 2db65230 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
root: INFO: * Waiting for engine to migrate...
lago.ssh: DEBUG: start task:f48ea071-64ea-48d4-b8a4-91c01f7d0bca:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:f48ea071-64ea-48d4-b8a4-91c01f7d0bca:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running 2e3e3952 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 2e3e3952 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 2e3e3952 on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3343 (Fri Sep 14 04:21:29 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3343 (Fri Sep 14 04:21:29 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "490d51f6", "local_conf_timestamp": 3343, "host-ts": 3343}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3341 (Fri Sep 14 04:21:27 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3341 (Fri Sep 14 04:21:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fae41c6b", "local_conf_timestamp": 3341, "host-ts": 3341}, "global_maintenance": false}
lago.ssh: DEBUG: start task:a395325a-1bb1-4628-afe9-3c044c03a15f:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:a395325a-1bb1-4628-afe9-3c044c03a15f:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running 34d2d908 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 34d2d908 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 34d2d908 on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3353 (Fri Sep 14 04:21:39 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3353 (Fri Sep 14 04:21:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7eb24832", "local_conf_timestamp": 3353, "host-ts": 3353}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3351 (Fri Sep 14 04:21:37 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3351 (Fri Sep 14 04:21:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c78129c7", "local_conf_timestamp": 3351, "host-ts": 3351}, "global_maintenance": false}
lago.ssh: DEBUG: start task:bcb68d9f-1dd3-4a4a-bc77-4dd79e76bbed:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:bcb68d9f-1dd3-4a4a-bc77-4dd79e76bbed:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running 372bbbf2 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 372bbbf2 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 372bbbf2 on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3353 (Fri Sep 14 04:21:39 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3353 (Fri Sep 14 04:21:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7eb24832", "local_conf_timestamp": 3353, "host-ts": 3353}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3351 (Fri Sep 14 04:21:37 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3351 (Fri Sep 14 04:21:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c78129c7", "local_conf_timestamp": 3351, "host-ts": 3351}, "global_maintenance": false}
lago.ssh: DEBUG: start task:09f83e32-8ee3-4d44-9da3-4223626e2ed9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:09f83e32-8ee3-4d44-9da3-4223626e2ed9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running 39875618 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 39875618 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 39875618 on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3353 (Fri Sep 14 04:21:39 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3353 (Fri Sep 14 04:21:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7eb24832", "local_conf_timestamp": 3353, "host-ts": 3353}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3351 (Fri Sep 14 04:21:37 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3351 (Fri Sep 14 04:21:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c78129c7", "local_conf_timestamp": 3351, "host-ts": 3351}, "global_maintenance": false}
lago.ssh: DEBUG: start task:cc21a5f2-c7d4-4df7-9212-efd5f0ddfb67:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:cc21a5f2-c7d4-4df7-9212-efd5f0ddfb67:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running 3c072cb0 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 3c072cb0 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 3c072cb0 on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3363 (Fri Sep 14 04:21:49 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3364 (Fri Sep 14 04:21:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d0ae2b36", "local_conf_timestamp": 3364, "host-ts": 3363}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3361 (Fri Sep 14 04:21:47 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3361 (Fri Sep 14 04:21:48 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "064376e1", "local_conf_timestamp": 3361, "host-ts": 3361}, "global_maintenance": false}
lago.ssh: DEBUG: start task:ed320820-2eb4-4618-ab75-f0607ad763c9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:ed320820-2eb4-4618-ab75-f0607ad763c9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running 3e643520 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 3e643520 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 3e643520 on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3363 (Fri Sep 14 04:21:49 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3364 (Fri Sep 14 04:21:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d0ae2b36", "local_conf_timestamp": 3364, "host-ts": 3363}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3361 (Fri Sep 14 04:21:47 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3361 (Fri Sep 14 04:21:48 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "064376e1", "local_conf_timestamp": 3361, "host-ts": 3361}, "global_maintenance": false}
lago.ssh: DEBUG: start task:5de6d339-db62-4214-8c40-42134ea48ff6:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:5de6d339-db62-4214-8c40-42134ea48ff6:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running 40baf412 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 40baf412 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 40baf412 on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1c53d696", "local_conf_timestamp": 3372, "host-ts": 3371}, "global_maintenance": false}
lago.ssh: DEBUG: start task:51240133-3070-4a87-9f8b-b059cf3f9f0b:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:51240133-3070-4a87-9f8b-b059cf3f9f0b:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running 43075d14 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 43075d14 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 43075d14 on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1c53d696", "local_conf_timestamp": 3372, "host-ts": 3371}, "global_maintenance": false}
lago.ssh: DEBUG: start task:e27a9a75-6f74-4c35-b53b-64e3224449b0:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: end task:e27a9a75-6f74-4c35-b53b-64e3224449b0:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:
lago.ssh: DEBUG: Running 4564db7c on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 4564db7c on lago-he-basic-ansible-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 4564db7c on lago-he-basic-ansible-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason
ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7fc20ea63a28>
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within
res = func()
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 36, in <lambda>
testlib.assert_true_within_long(lambda: _get_he_status(host)
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 98, in _get_he_status
raise RuntimeError('could not parse JSON: %s' % ret.out)
RuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason
--------------------- >> end captured logging << ---------------------
Stack Trace:
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 125, in local_maintenance
_wait_for_engine_migration(host, he_index, "bad", "Migration Destination")
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 36, in _wait_for_engine_migration
testlib.assert_true_within_long(lambda: _get_he_status(host)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long
assert_equals_within_long(func, True, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long
func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within
res = func()
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 36, in <lambda>
testlib.assert_true_within_long(lambda: _get_he_status(host)
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 98, in _get_he_status
raise RuntimeError('could not parse JSON: %s' % ret.out)
'could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nlago.ssh: DEBUG: start task:0874be8a-4d1f-4242-8c58-343d3bf8718a:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:0874be8a-4d1f-4242-8c58-343d3bf8718a:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 2d38dc7e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 2d38dc7e on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 2d38dc7e on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3333 (Fri Sep 14 04:21:19 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=3333 (Fri Sep 14 04:21:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b74983cd", "local_conf_timestamp": 3333, "host-ts": 3333}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3331 (Fri Sep 14 04:21:18 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3331 (Fri Sep 14 04:21:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "547f6cbb", "local_conf_timestamp": 3331, "host-ts": 3331}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:de4315ba-d73d-4a51-8315-8f960c2a0e35:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:de4315ba-d73d-4a51-8315-8f960c2a0e35:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 2db65230 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local\nlago.ssh: DEBUG: Command 2db65230 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nroot: INFO: * Waiting for engine to migrate...\nlago.ssh: DEBUG: start task:f48ea071-64ea-48d4-b8a4-91c01f7d0bca:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:f48ea071-64ea-48d4-b8a4-91c01f7d0bca:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 2e3e3952 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 2e3e3952 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 2e3e3952 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3343 (Fri Sep 14 04:21:29 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3343 (Fri Sep 14 04:21:29 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "490d51f6", "local_conf_timestamp": 3343, "host-ts": 3343}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3341 (Fri Sep 14 04:21:27 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3341 (Fri Sep 14 04:21:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fae41c6b", "local_conf_timestamp": 3341, "host-ts": 3341}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:a395325a-1bb1-4628-afe9-3c044c03a15f:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:a395325a-1bb1-4628-afe9-3c044c03a15f:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 34d2d908 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 34d2d908 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 34d2d908 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3353 (Fri Sep 14 04:21:39 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3353 (Fri Sep 14 04:21:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7eb24832", "local_conf_timestamp": 3353, "host-ts": 3353}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3351 (Fri Sep 14 04:21:37 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3351 (Fri Sep 14 04:21:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c78129c7", "local_conf_timestamp": 3351, "host-ts": 3351}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:bcb68d9f-1dd3-4a4a-bc77-4dd79e76bbed:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:bcb68d9f-1dd3-4a4a-bc77-4dd79e76bbed:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 372bbbf2 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 372bbbf2 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 372bbbf2 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3353 (Fri Sep 14 04:21:39 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3353 (Fri Sep 14 04:21:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7eb24832", "local_conf_timestamp": 3353, "host-ts": 3353}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3351 (Fri Sep 14 04:21:37 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3351 (Fri Sep 14 04:21:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c78129c7", "local_conf_timestamp": 3351, "host-ts": 3351}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:09f83e32-8ee3-4d44-9da3-4223626e2ed9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:09f83e32-8ee3-4d44-9da3-4223626e2ed9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 39875618 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 39875618 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 39875618 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3353 (Fri Sep 14 04:21:39 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3353 (Fri Sep 14 04:21:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7eb24832", "local_conf_timestamp": 3353, "host-ts": 3353}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3351 (Fri Sep 14 04:21:37 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3351 (Fri Sep 14 04:21:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c78129c7", "local_conf_timestamp": 3351, "host-ts": 3351}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:cc21a5f2-c7d4-4df7-9212-efd5f0ddfb67:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:cc21a5f2-c7d4-4df7-9212-efd5f0ddfb67:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 3c072cb0 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 3c072cb0 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 3c072cb0 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3363 (Fri Sep 14 04:21:49 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3364 (Fri Sep 14 04:21:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d0ae2b36", "local_conf_timestamp": 3364, "host-ts": 3363}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3361 (Fri Sep 14 04:21:47 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3361 (Fri Sep 14 04:21:48 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "064376e1", "local_conf_timestamp": 3361, "host-ts": 3361}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:ed320820-2eb4-4618-ab75-f0607ad763c9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:ed320820-2eb4-4618-ab75-f0607ad763c9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 3e643520 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 3e643520 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 3e643520 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3363 (Fri Sep 14 04:21:49 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3364 (Fri Sep 14 04:21:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d0ae2b36", "local_conf_timestamp": 3364, "host-ts": 3363}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3361 (Fri Sep 14 04:21:47 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3361 (Fri Sep 14 04:21:48 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "064376e1", "local_conf_timestamp": 3361, "host-ts": 3361}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:5de6d339-db62-4214-8c40-42134ea48ff6:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:5de6d339-db62-4214-8c40-42134ea48ff6:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 40baf412 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 40baf412 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 40baf412 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1c53d696", "local_conf_timestamp": 3372, "host-ts": 3371}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:51240133-3070-4a87-9f8b-b059cf3f9f0b:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:51240133-3070-4a87-9f8b-b059cf3f9f0b:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 43075d14 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 43075d14 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 43075d14 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1c53d696", "local_conf_timestamp": 3372, "host-ts": 3371}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:e27a9a75-6f74-4c35-b53b-64e3224449b0:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:e27a9a75-6f74-4c35-b53b-64e3224449b0:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 4564db7c on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 4564db7c on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 4564db7c on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7fc20ea63a28>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 36, in <lambda>\n testlib.assert_true_within_long(lambda: _get_he_status(host)\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 98, in _get_he_status\n raise RuntimeError(\'could not parse JSON: %s\' % ret.out)\nRuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason\n--------------------- >> end captured logging << ---------------------'
6 years, 2 months
[JIRA] (OVIRT-2498) Failing KubeVirt CI
by Petr Kotas (oVirt JIRA)
Petr Kotas created OVIRT-2498:
---------------------------------
Summary: Failing KubeVirt CI
Key: OVIRT-2498
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2498
Project: oVirt - virtualization made easy
Issue Type: By-EMAIL
Reporter: Petr Kotas
Assignee: infra
Hi,
I am working on fixing the issues on the KubeVirt e2e test suites. This
task is directly related to unstable CI, due to unknown errors.
The progress is reported in the CNV trello:
https://trello.com/c/HNXcMEQu/161-epic-improve-ci
I am creating this issue since the KubeVirt experience random timeouts on
random tests most of the times when test suites run.
The issue from outside is showing as timeouts on difference part of tests.
Sometimes the tests fails in set up phase, again due to random timeout.
The example in the link bellow timed out for network connection on
localhost.
[check-patch.k8s-1.11.0-dev.el7.x86_64]
requests.exceptions.ReadTimeout:
UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.
(read timeout=60)
Example of failing test suites is here
https://jenkins.ovirt.org/job/kubevirt_kubevirt_standard-check-pr/1916/co...
The list of errors related to the failing CI can be found in my notes
https://docs.google.com/document/d/1_ll1DOMHgCRHn_Df9i4uvtRFyMK-bDCHEeGfJ...
I am not sure whether KubeVirt already shared the resource requirements, so
I provide short summary:
*Resources for KubeVirt e2e tests:*
- at least 12GB of RAM - we start 3 nodes (3 docker images) each require
4GB of RAM
- exposed /dev/kvm to enable native virtualization
- cached images, since these are used to build the test cluster:
- kubevirtci/os-3.10.0-crio:latest
- kubevirtci/os-3.10.0-multus:latest
- kubevirtci/os-3.10.0:latest
- kubevirtci/k8s-1.10.4:latest
- kubevirtci/k8s-multus-1.11.1:latest
- kubevirtci/k8s-1.11.0:latest
How can we overcome this? Can we work together to build a suitable
requirements for running the tests so it passes each time?
Kind regards,
Petr Kotas
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100092)
6 years, 2 months
[JIRA] (OVIRT-2494) CQ Failure - Cannot add host
by Dafna Ron (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2494?page=com.atlassian.jir... ]
Dafna Ron updated OVIRT-2494:
-----------------------------
Issue Type: Bug (was: Outage)
> CQ Failure - Cannot add host
> ----------------------------
>
> Key: OVIRT-2494
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2494
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Components: Change Queue, OST
> Reporter: Ehud Yonasi
> Assignee: infra
> Labels: cq-issue-traking, ost_failures
>
> Hi,
> This is an example of a he basic suite master that fails on :
> Error Message:
> 00:26:36 Error while running thread
> 00:26:36 Traceback (most recent call last):
> 00:26:36 File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue
> 00:26:36 queue.put({'return': func()})
> 00:26:36 File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-master/ovirt-system-tests/he-basic-suite-master/test-scenarios/002_bootstrap.py", line 379, in _add_he_host_4
> 00:26:36 deploy_hosted_engine=True,
> 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 11530, in add
> 00:26:36 return self._internal_add(host, headers, query, wait)
> 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add
> 00:26:36 return future.wait() if wait else future
> 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait
> 00:26:36 return self._code(response)
> 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback
> 00:26:36 self._check_fault(response)
> 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault
> 00:26:36 self._raise_error(response, body)
> 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error
> 00:26:36 raise error
> 00:26:36 Error: Fault reason is "Operation Failed". Fault detail is "[Cannot add Host. You are using an unmanaged hosted engine VM. Please add the first storage domain in order to start the hosted engine import process.]". HTTP response code is 409.
> Job url:
> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te...
> There is a patch which was merged:
> https://gerrit.ovirt.org/#/c/94219/
> It did not pass CQ because of another failure which Ravi is working on ( Add cluster issue).
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100092)
6 years, 2 months