September 2018 - Infra - oVirt List Archives

[oVirt Jenkins] ovirt-system-tests_he-basic-ansible-suite-4.2 - Build # 565 - Failure!
by jenkins＠jenkins.phx.ovirt.org 14 Sep '18

14 Sep '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4.2/… Build Number: 565 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #565 [Milan Zamazal] Update test-vm.ovf file ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 010_local_maintenance_cli.local_maintenance Error Message: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... lago.ssh: DEBUG: start task:0874be8a-4d1f-4242-8c58-343d3bf8718a:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:0874be8a-4d1f-4242-8c58-343d3bf8718a:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 2d38dc7e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 2d38dc7e on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 2d38dc7e on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3333 (Fri Sep 14 04:21:19 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=3333 (Fri Sep 14 04:21:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b74983cd", "local_conf_timestamp": 3333, "host-ts": 3333}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3331 (Fri Sep 14 04:21:18 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3331 (Fri Sep 14 04:21:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "547f6cbb", "local_conf_timestamp": 3331, "host-ts": 3331}, "global_maintenance": false} lago.ssh: DEBUG: start task:de4315ba-d73d-4a51-8315-8f960c2a0e35:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:de4315ba-d73d-4a51-8315-8f960c2a0e35:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 2db65230 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local lago.ssh: DEBUG: Command 2db65230 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 root: INFO: * Waiting for engine to migrate... lago.ssh: DEBUG: start task:f48ea071-64ea-48d4-b8a4-91c01f7d0bca:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:f48ea071-64ea-48d4-b8a4-91c01f7d0bca:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 2e3e3952 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 2e3e3952 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 2e3e3952 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3343 (Fri Sep 14 04:21:29 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3343 (Fri Sep 14 04:21:29 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "490d51f6", "local_conf_timestamp": 3343, "host-ts": 3343}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3341 (Fri Sep 14 04:21:27 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3341 (Fri Sep 14 04:21:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fae41c6b", "local_conf_timestamp": 3341, "host-ts": 3341}, "global_maintenance": false} lago.ssh: DEBUG: start task:a395325a-1bb1-4628-afe9-3c044c03a15f:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:a395325a-1bb1-4628-afe9-3c044c03a15f:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 34d2d908 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 34d2d908 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 34d2d908 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3353 (Fri Sep 14 04:21:39 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3353 (Fri Sep 14 04:21:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7eb24832", "local_conf_timestamp": 3353, "host-ts": 3353}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3351 (Fri Sep 14 04:21:37 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3351 (Fri Sep 14 04:21:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c78129c7", "local_conf_timestamp": 3351, "host-ts": 3351}, "global_maintenance": false} lago.ssh: DEBUG: start task:bcb68d9f-1dd3-4a4a-bc77-4dd79e76bbed:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:bcb68d9f-1dd3-4a4a-bc77-4dd79e76bbed:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 372bbbf2 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 372bbbf2 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 372bbbf2 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3353 (Fri Sep 14 04:21:39 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3353 (Fri Sep 14 04:21:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7eb24832", "local_conf_timestamp": 3353, "host-ts": 3353}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3351 (Fri Sep 14 04:21:37 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3351 (Fri Sep 14 04:21:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c78129c7", "local_conf_timestamp": 3351, "host-ts": 3351}, "global_maintenance": false} lago.ssh: DEBUG: start task:09f83e32-8ee3-4d44-9da3-4223626e2ed9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:09f83e32-8ee3-4d44-9da3-4223626e2ed9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 39875618 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 39875618 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 39875618 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3353 (Fri Sep 14 04:21:39 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3353 (Fri Sep 14 04:21:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7eb24832", "local_conf_timestamp": 3353, "host-ts": 3353}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3351 (Fri Sep 14 04:21:37 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3351 (Fri Sep 14 04:21:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c78129c7", "local_conf_timestamp": 3351, "host-ts": 3351}, "global_maintenance": false} lago.ssh: DEBUG: start task:cc21a5f2-c7d4-4df7-9212-efd5f0ddfb67:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:cc21a5f2-c7d4-4df7-9212-efd5f0ddfb67:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 3c072cb0 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 3c072cb0 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 3c072cb0 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3363 (Fri Sep 14 04:21:49 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3364 (Fri Sep 14 04:21:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d0ae2b36", "local_conf_timestamp": 3364, "host-ts": 3363}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3361 (Fri Sep 14 04:21:47 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3361 (Fri Sep 14 04:21:48 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "064376e1", "local_conf_timestamp": 3361, "host-ts": 3361}, "global_maintenance": false} lago.ssh: DEBUG: start task:ed320820-2eb4-4618-ab75-f0607ad763c9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:ed320820-2eb4-4618-ab75-f0607ad763c9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 3e643520 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 3e643520 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 3e643520 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3363 (Fri Sep 14 04:21:49 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3364 (Fri Sep 14 04:21:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d0ae2b36", "local_conf_timestamp": 3364, "host-ts": 3363}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3361 (Fri Sep 14 04:21:47 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3361 (Fri Sep 14 04:21:48 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "064376e1", "local_conf_timestamp": 3361, "host-ts": 3361}, "global_maintenance": false} lago.ssh: DEBUG: start task:5de6d339-db62-4214-8c40-42134ea48ff6:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:5de6d339-db62-4214-8c40-42134ea48ff6:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 40baf412 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 40baf412 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 40baf412 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1c53d696", "local_conf_timestamp": 3372, "host-ts": 3371}, "global_maintenance": false} lago.ssh: DEBUG: start task:51240133-3070-4a87-9f8b-b059cf3f9f0b:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:51240133-3070-4a87-9f8b-b059cf3f9f0b:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 43075d14 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 43075d14 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 43075d14 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1c53d696", "local_conf_timestamp": 3372, "host-ts": 3371}, "global_maintenance": false} lago.ssh: DEBUG: start task:e27a9a75-6f74-4c35-b53b-64e3224449b0:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:e27a9a75-6f74-4c35-b53b-64e3224449b0:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 4564db7c on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 4564db7c on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 4564db7c on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7fc20ea63a28> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 36, in <lambda> testlib.assert_true_within_long(lambda: _get_he_status(host) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 98, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) RuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 125, in local_maintenance _wait_for_engine_migration(host, he_index, "bad", "Migration Destination") File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 36, in _wait_for_engine_migration testlib.assert_true_within_long(lambda: _get_he_status(host) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 36, in <lambda> testlib.assert_true_within_long(lambda: _get_he_status(host) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 98, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) 'could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nlago.ssh: DEBUG: start task:0874be8a-4d1f-4242-8c58-343d3bf8718a:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:0874be8a-4d1f-4242-8c58-343d3bf8718a:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 2d38dc7e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 2d38dc7e on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 2d38dc7e on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3333 (Fri Sep 14 04:21:19 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=3333 (Fri Sep 14 04:21:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b74983cd", "local_conf_timestamp": 3333, "host-ts": 3333}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3331 (Fri Sep 14 04:21:18 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3331 (Fri Sep 14 04:21:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "547f6cbb", "local_conf_timestamp": 3331, "host-ts": 3331}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:de4315ba-d73d-4a51-8315-8f960c2a0e35:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:de4315ba-d73d-4a51-8315-8f960c2a0e35:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 2db65230 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local\nlago.ssh: DEBUG: Command 2db65230 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nroot: INFO: * Waiting for engine to migrate...\nlago.ssh: DEBUG: start task:f48ea071-64ea-48d4-b8a4-91c01f7d0bca:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:f48ea071-64ea-48d4-b8a4-91c01f7d0bca:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 2e3e3952 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 2e3e3952 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 2e3e3952 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3343 (Fri Sep 14 04:21:29 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3343 (Fri Sep 14 04:21:29 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "490d51f6", "local_conf_timestamp": 3343, "host-ts": 3343}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3341 (Fri Sep 14 04:21:27 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3341 (Fri Sep 14 04:21:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fae41c6b", "local_conf_timestamp": 3341, "host-ts": 3341}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:a395325a-1bb1-4628-afe9-3c044c03a15f:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:a395325a-1bb1-4628-afe9-3c044c03a15f:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 34d2d908 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 34d2d908 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 34d2d908 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3353 (Fri Sep 14 04:21:39 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3353 (Fri Sep 14 04:21:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7eb24832", "local_conf_timestamp": 3353, "host-ts": 3353}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3351 (Fri Sep 14 04:21:37 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3351 (Fri Sep 14 04:21:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c78129c7", "local_conf_timestamp": 3351, "host-ts": 3351}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:bcb68d9f-1dd3-4a4a-bc77-4dd79e76bbed:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:bcb68d9f-1dd3-4a4a-bc77-4dd79e76bbed:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 372bbbf2 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 372bbbf2 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 372bbbf2 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3353 (Fri Sep 14 04:21:39 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3353 (Fri Sep 14 04:21:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7eb24832", "local_conf_timestamp": 3353, "host-ts": 3353}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3351 (Fri Sep 14 04:21:37 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3351 (Fri Sep 14 04:21:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c78129c7", "local_conf_timestamp": 3351, "host-ts": 3351}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:09f83e32-8ee3-4d44-9da3-4223626e2ed9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:09f83e32-8ee3-4d44-9da3-4223626e2ed9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 39875618 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 39875618 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 39875618 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3353 (Fri Sep 14 04:21:39 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3353 (Fri Sep 14 04:21:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7eb24832", "local_conf_timestamp": 3353, "host-ts": 3353}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3351 (Fri Sep 14 04:21:37 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3351 (Fri Sep 14 04:21:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c78129c7", "local_conf_timestamp": 3351, "host-ts": 3351}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:cc21a5f2-c7d4-4df7-9212-efd5f0ddfb67:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:cc21a5f2-c7d4-4df7-9212-efd5f0ddfb67:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 3c072cb0 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 3c072cb0 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 3c072cb0 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3363 (Fri Sep 14 04:21:49 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3364 (Fri Sep 14 04:21:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d0ae2b36", "local_conf_timestamp": 3364, "host-ts": 3363}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3361 (Fri Sep 14 04:21:47 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3361 (Fri Sep 14 04:21:48 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "064376e1", "local_conf_timestamp": 3361, "host-ts": 3361}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:ed320820-2eb4-4618-ab75-f0607ad763c9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:ed320820-2eb4-4618-ab75-f0607ad763c9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 3e643520 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 3e643520 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 3e643520 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3363 (Fri Sep 14 04:21:49 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3364 (Fri Sep 14 04:21:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d0ae2b36", "local_conf_timestamp": 3364, "host-ts": 3363}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3361 (Fri Sep 14 04:21:47 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3361 (Fri Sep 14 04:21:48 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "064376e1", "local_conf_timestamp": 3361, "host-ts": 3361}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:5de6d339-db62-4214-8c40-42134ea48ff6:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:5de6d339-db62-4214-8c40-42134ea48ff6:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 40baf412 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 40baf412 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 40baf412 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1c53d696", "local_conf_timestamp": 3372, "host-ts": 3371}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:51240133-3070-4a87-9f8b-b059cf3f9f0b:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:51240133-3070-4a87-9f8b-b059cf3f9f0b:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 43075d14 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 43075d14 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 43075d14 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1c53d696", "local_conf_timestamp": 3372, "host-ts": 3371}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:e27a9a75-6f74-4c35-b53b-64e3224449b0:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:e27a9a75-6f74-4c35-b53b-64e3224449b0:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 4564db7c on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 4564db7c on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 4564db7c on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7fc20ea63a28>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 36, in <lambda>\n testlib.assert_true_within_long(lambda: _get_he_status(host)\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 98, in _get_he_status\n raise RuntimeError(\'could not parse JSON: %s\' % ret.out)\nRuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason\n--------------------- >> end captured logging << ---------------------'

1 1

[CQ]: 94344, 1 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 14 Sep '18

14 Sep '18

A system test invoked by the "ovirt-master" change queue including change 94344,1 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 94081,3 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 94081,3 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/94344/1 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/94081/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10314/

1 0

[JIRA] (OVIRT-2498) Failing KubeVirt CI
by Petr Kotas (oVirt JIRA) 14 Sep '18

14 Sep '18

Petr Kotas created OVIRT-2498: --------------------------------- Summary: Failing KubeVirt CI Key: OVIRT-2498 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2498 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Petr Kotas Assignee: infra Hi, I am working on fixing the issues on the KubeVirt e2e test suites. This task is directly related to unstable CI, due to unknown errors. The progress is reported in the CNV trello: https://trello.com/c/HNXcMEQu/161-epic-improve-ci I am creating this issue since the KubeVirt experience random timeouts on random tests most of the times when test suites run. The issue from outside is showing as timeouts on difference part of tests. Sometimes the tests fails in set up phase, again due to random timeout. The example in the link bellow timed out for network connection on localhost. [check-patch.k8s-1.11.0-dev.el7.x86_64] requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60) Example of failing test suites is here https://jenkins.ovirt.org/job/kubevirt_kubevirt_standard-check-pr/1916/cons… The list of errors related to the failing CI can be found in my notes https://docs.google.com/document/d/1_ll1DOMHgCRHn_Df9i4uvtRFyMK-bDCHEeGfJFT… I am not sure whether KubeVirt already shared the resource requirements, so I provide short summary: *Resources for KubeVirt e2e tests:* - at least 12GB of RAM - we start 3 nodes (3 docker images) each require 4GB of RAM - exposed /dev/kvm to enable native virtualization - cached images, since these are used to build the test cluster: - kubevirtci/os-3.10.0-crio:latest - kubevirtci/os-3.10.0-multus:latest - kubevirtci/os-3.10.0:latest - kubevirtci/k8s-1.10.4:latest - kubevirtci/k8s-multus-1.11.1:latest - kubevirtci/k8s-1.11.0:latest How can we overcome this? Can we work together to build a suitable requirements for running the tests so it passes each time? Kind regards, Petr Kotas -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[CQ]: 94317, 2 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 14 Sep '18

14 Sep '18

A system test invoked by the "ovirt-master" change queue including change 94317,2 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 94249,9 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 94249,9 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/94317/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/94249/9 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10311/

2 1

[CQ]: 5650929 (ovirt-ansible-v2v-conversion-host) failed "ovirt-4.2" system tests
by oVirt Jenkins 14 Sep '18

14 Sep '18

Change 5650929 (ovirt-ansible-v2v-conversion-host) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://github.com/oVirt/ovirt-ansible-v2v-conversion-host/commit/565092928… For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3112/

2 1

[JIRA] (OVIRT-2497) ovirt-master in project vdsm failing on migrate vm test
by Dafna Ron (oVirt JIRA) 14 Sep '18

14 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2497?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2497: ---------------------------------- the patch that is root cause may actually be the cause because its part of the change. but it can also be the api patch. I added a comment in the mail I sent to dev but forgot to update the ticket > ovirt-master in project vdsm failing on migrate vm test > ------------------------------------------------------- > > Key: OVIRT-2497 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2497 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_code_regression, ost_failures > > we have a failure in migrate vm for project vdsm > full logs can be found here: > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10309/artifa… > reported patch and root cause patch do not seem related to the failure. > I suspect this is the change that caused the issue: > https://gerrit.ovirt.org/#/c/94248/ > error: https://pastebin.com/XNGFgfvL -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2497) ovirt-master in project vdsm failing on migrate vm test
by Barak Korren (oVirt JIRA) 14 Sep '18

14 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2497?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-2497: ------------------------------------- בתאריך יום ו׳, 14 בספט׳ 2018, 15:20, מאת Dafna Ron (oVirt JIRA) ‏< So why did the root cause patch fail? I suspect this is the change that caused the issue: > ovirt-master in project vdsm failing on migrate vm test > ------------------------------------------------------- > > Key: OVIRT-2497 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2497 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_code_regression, ost_failures > > we have a failure in migrate vm for project vdsm > full logs can be found here: > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10309/artifa… > reported patch and root cause patch do not seem related to the failure. > I suspect this is the change that caused the issue: > https://gerrit.ovirt.org/#/c/94248/ > error: https://pastebin.com/XNGFgfvL -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2497) ovirt-master in project vdsm failing on migrate vm test
by Dafna Ron (oVirt JIRA) 14 Sep '18

14 Sep '18

Dafna Ron created OVIRT-2497: -------------------------------- Summary: ovirt-master in project vdsm failing on migrate vm test Key: OVIRT-2497 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2497 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Dafna Ron Assignee: infra Priority: Highest we have a failure in migrate vm for project vdsm full logs can be found here: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10309/artifa… reported patch and root cause patch do not seem related to the failure. I suspect this is the change that caused the issue: https://gerrit.ovirt.org/#/c/94248/ error: https://pastebin.com/XNGFgfvL -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

2 1

[CQ]: 94333, 2 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 14 Sep '18

14 Sep '18

A system test invoked by the "ovirt-master" change queue including change 94333,2 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 94249,9 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 94249,9 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/94333/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/94249/9 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10309/

2 1

[JIRA] (OVIRT-2494) CQ Failure - Cannot add host
by Dafna Ron (oVirt JIRA) 14 Sep '18

14 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2494?page=com.atlassian.jira.… ] Dafna Ron updated OVIRT-2494: ----------------------------- Issue Type: Bug (was: Outage) > CQ Failure - Cannot add host > ---------------------------- > > Key: OVIRT-2494 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2494 > Project: oVirt - virtualization made easy > Issue Type: Bug > Components: Change Queue, OST > Reporter: Ehud Yonasi > Assignee: infra > Labels: cq-issue-traking, ost_failures > > Hi, > This is an example of a he basic suite master that fails on : > Error Message: > 00:26:36 Error while running thread > 00:26:36 Traceback (most recent call last): > 00:26:36 File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue > 00:26:36 queue.put({'return': func()}) > 00:26:36 File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-master/ovirt-system-tests/he-basic-suite-master/test-scenarios/002_bootstrap.py", line 379, in _add_he_host_4 > 00:26:36 deploy_hosted_engine=True, > 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 11530, in add > 00:26:36 return self._internal_add(host, headers, query, wait) > 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add > 00:26:36 return future.wait() if wait else future > 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait > 00:26:36 return self._code(response) > 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback > 00:26:36 self._check_fault(response) > 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault > 00:26:36 self._raise_error(response, body) > 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error > 00:26:36 raise error > 00:26:36 Error: Fault reason is "Operation Failed". Fault detail is "[Cannot add Host. You are using an unmanaged hosted engine VM. Please add the first storage domain in order to start the hosted engine import process.]". HTTP response code is 409. > Job url: > https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… > There is a patch which was merged: > https://gerrit.ovirt.org/#/c/94219/ > It did not pass CQ because of another failure which Ravi is working on ( Add cluster issue). -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0