[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-4.2 - Build # 285 - Failure!
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/285/ Build Number: 285 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #285 [Gal Ben Haim] Adding dr suite [Barak Korren] Remove the populate_mock function from mock_runner [Daniel Belenky] mock_runner: store shell cmd in a variable [Barak Korren] mock_runner: Added timeout param [Barak Korren] Make whitelist repo configurable via env vars [Daniel Belenky] stdci_runner: let mock_runner manage timeout [Greg Sheremeta] remove 4.1 change queue from ovirt-engine-nodejs config ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 010_local_maintenance_cli.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... lago.ssh: DEBUG: start task:c3e22bc1-2785-476e-8ba3-ad975b3823a0:Get ssh client for lago-he-node-ng-suite-4-2-host-1: lago.ssh: DEBUG: end task:c3e22bc1-2785-476e-8ba3-ad975b3823a0:Get ssh client for lago-he-node-ng-suite-4-2-host-1: lago.ssh: DEBUG: Running 4cc65a9e on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 4cc65a9e on lago-he-node-ng-suite-4-2-host-1 returned with 0 lago.ssh: DEBUG: Command 4cc65a9e on lago-he-node-ng-suite-4-2-host-1 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4109 (Thu Sep 6 01:43:37 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4109 (Thu Sep 6 01:43:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "044228a8", "local_conf_timestamp": 4109, "host-ts": 4109}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4115 (Thu Sep 6 01:43:42 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4115 (Thu Sep 6 01:43:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8d059996", "local_conf_timestamp": 4115, "host-ts": 4115}, "global_maintenance": false} lago.ssh: DEBUG: start task:7f537510-6e18-4173-b0f5-d5f6a1c6e460:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:7f537510-6e18-4173-b0f5-d5f6a1c6e460:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 4dde7bbe on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local lago.ssh: DEBUG: Command 4dde7bbe on lago-he-node-ng-suite-4-2-host-0 returned with 0 root: INFO: * Waiting for engine to migrate... lago.ssh: DEBUG: start task:94120632-275d-478f-9cac-0ff1eb6c2e63:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:94120632-275d-478f-9cac-0ff1eb6c2e63:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 4ef81abe on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 4ef81abe on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 4ef81abe on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4119 (Thu Sep 6 01:43:47 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4120 (Thu Sep 6 01:43:47 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8ad483f0", "local_conf_timestamp": 4120, "host-ts": 4119}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4115 (Thu Sep 6 01:43:42 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4115 (Thu Sep 6 01:43:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8d059996", "local_conf_timestamp": 4115, "host-ts": 4115}, "global_maintenance": false} lago.ssh: DEBUG: start task:82d6fe01-3354-4477-9210-df6ac2d4d420:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:82d6fe01-3354-4477-9210-df6ac2d4d420:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 5626eb30 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 5626eb30 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 5626eb30 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4131 (Thu Sep 6 01:43:58 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4130 (Thu Sep 6 01:43:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "65ca1ba5", "local_conf_timestamp": 4130, "host-ts": 4131}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4125 (Thu Sep 6 01:43:53 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4126 (Thu Sep 6 01:43:53 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "18c7c923", "local_conf_timestamp": 4126, "host-ts": 4125}, "global_maintenance": false} lago.ssh: DEBUG: start task:b3211ae7-4af2-469b-92c0-38ab6f07c5c4:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:b3211ae7-4af2-469b-92c0-38ab6f07c5c4:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 59979f1c on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 59979f1c on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 59979f1c on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4141 (Thu Sep 6 01:44:09 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4141 (Thu Sep 6 01:44:09 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "49290bc0", "local_conf_timestamp": 4141, "host-ts": 4141}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4135 (Thu Sep 6 01:44:02 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4135 (Thu Sep 6 01:44:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8588679c", "local_conf_timestamp": 4135, "host-ts": 4135}, "global_maintenance": false} lago.ssh: DEBUG: start task:f62305f7-c324-4ee2-ba11-d092ac823600:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:f62305f7-c324-4ee2-ba11-d092ac823600:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 5dfd4692 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 5dfd4692 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 5dfd4692 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4151 (Thu Sep 6 01:44:18 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4151 (Thu Sep 6 01:44:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e71a3502", "local_conf_timestamp": 4151, "host-ts": 4151}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4145 (Thu Sep 6 01:44:13 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4145 (Thu Sep 6 01:44:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ef65af1f", "local_conf_timestamp": 4145, "host-ts": 4145}, "global_maintenance": false} lago.ssh: DEBUG: start task:a553c343-1311-45f3-9fc3-ed53d93b21d5:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:a553c343-1311-45f3-9fc3-ed53d93b21d5:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 61aded00 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 61aded00 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 61aded00 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4151 (Thu Sep 6 01:44:18 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4151 (Thu Sep 6 01:44:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e71a3502", "local_conf_timestamp": 4151, "host-ts": 4151}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4145 (Thu Sep 6 01:44:13 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4145 (Thu Sep 6 01:44:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ef65af1f", "local_conf_timestamp": 4145, "host-ts": 4145}, "global_maintenance": false} lago.ssh: DEBUG: start task:bb63e900-36b1-47d5-8bd7-b6b67eed6f7e:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:bb63e900-36b1-47d5-8bd7-b6b67eed6f7e:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 64b1136a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 64b1136a on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 64b1136a on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4151 (Thu Sep 6 01:44:18 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4151 (Thu Sep 6 01:44:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e71a3502", "local_conf_timestamp": 4151, "host-ts": 4151}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4155 (Thu Sep 6 01:44:22 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4155 (Thu Sep 6 01:44:22 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eb23501a", "local_conf_timestamp": 4155, "host-ts": 4155}, "global_maintenance": false} lago.ssh: DEBUG: start task:97447141-6b3c-4188-ad5b-a8bbe1cc45db:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:97447141-6b3c-4188-ad5b-a8bbe1cc45db:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 67a5dc18 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 67a5dc18 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 67a5dc18 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4161 (Thu Sep 6 01:44:29 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4162 (Thu Sep 6 01:44:30 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "573721bf", "local_conf_timestamp": 4162, "host-ts": 4161}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4155 (Thu Sep 6 01:44:22 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4155 (Thu Sep 6 01:44:22 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eb23501a", "local_conf_timestamp": 4155, "host-ts": 4155}, "global_maintenance": false} lago.ssh: DEBUG: start task:dcbacc88-4c96-4575-a53a-31b073b1e31e:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:dcbacc88-4c96-4575-a53a-31b073b1e31e:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 6ada63f4 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 6ada63f4 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 6ada63f4 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4171 (Thu Sep 6 01:44:39 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4172 (Thu Sep 6 01:44:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f9041f7d", "local_conf_timestamp": 4172, "host-ts": 4171}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4165 (Thu Sep 6 01:44:32 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4165 (Thu Sep 6 01:44:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "54cc899b", "local_conf_timestamp": 4165, "host-ts": 4165}, "global_maintenance": false} lago.ssh: DEBUG: start task:90045a52-c293-4837-bfef-33a5a1910dc5:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:90045a52-c293-4837-bfef-33a5a1910dc5:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 6df25e2a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 6df25e2a on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 6df25e2a on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4171 (Thu Sep 6 01:44:39 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4172 (Thu Sep 6 01:44:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f9041f7d", "local_conf_timestamp": 4172, "host-ts": 4171}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4165 (Thu Sep 6 01:44:32 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4165 (Thu Sep 6 01:44:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "54cc899b", "local_conf_timestamp": 4165, "host-ts": 4165}, "global_maintenance": false} lago.ssh: DEBUG: start task:a7476663-b52b-4275-8cc2-cd9ff3b88a3c:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:a7476663-b52b-4275-8cc2-cd9ff3b88a3c:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 71077262 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 71077262 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 71077262 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4181 (Thu Sep 6 01:44:48 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4181 (Thu Sep 6 01:44:49 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "edb81fdc", "local_conf_timestamp": 4181, "host-ts": 4181}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4175 (Thu Sep 6 01:44:43 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4176 (Thu Sep 6 01:44:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "c9832724", "local_conf_timestamp": 4176, "host-ts": 4175}, "global_maintenance": false} lago.ssh: DEBUG: start task:42a25e42-c90a-456e-8669-82df79c7507c:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:42a25e42-c90a-456e-8669-82df79c7507c:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 74595c8c on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 74595c8c on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 74595c8c on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4181 (Thu Sep 6 01:44:48 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4181 (Thu Sep 6 01:44:49 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "edb81fdc", "local_conf_timestamp": 4181, "host-ts": 4181}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4175 (Thu Sep 6 01:44:43 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4176 (Thu Sep 6 01:44:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "c9832724", "local_conf_timestamp": 4176, "host-ts": 4175}, "global_maintenance": false} lago.ssh: DEBUG: start task:22a966cf-52a3-43b7-98b0-a28f0df8f67e:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:22a966cf-52a3-43b7-98b0-a28f0df8f67e:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 77bc4bdc on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 77bc4bdc on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 77bc4bdc on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4191 (Thu Sep 6 01:44:59 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4192 (Thu Sep 6 01:45:00 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "VM migrated away successfully", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "29ba0d11", "local_conf_timestamp": 4192, "host-ts": 4191}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4185 (Thu Sep 6 01:44:52 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4185 (Thu Sep 6 01:44:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "6f6d73ad", "local_conf_timestamp": 4185, "host-ts": 4185}, "global_maintenance": false} lago.ssh: DEBUG: start task:f9fa6fdb-2c5a-41b9-ac58-6ec7a8e8ef3f:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:f9fa6fdb-2c5a-41b9-ac58-6ec7a8e8ef3f:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 78d063fa on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 78d063fa on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 78d063fa on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4191 (Thu Sep 6 01:44:59 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4192 (Thu Sep 6 01:45:00 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "VM migrated away successfully", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "29ba0d11", "local_conf_timestamp": 4192, "host-ts": 4191}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4185 (Thu Sep 6 01:44:52 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4185 (Thu Sep 6 01:44:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "6f6d73ad", "local_conf_timestamp": 4185, "host-ts": 4185}, "global_maintenance": false} lago.ssh: DEBUG: start task:095cd3ad-d465-46bb-8cd1-fd07a1a124ed:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:095cd3ad-d465-46bb-8cd1-fd07a1a124ed:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 7fdb3a94 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 7fdb3a94 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 7fdb3a94 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4202 (Thu Sep 6 01:45:10 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=4203 (Thu Sep 6 01:45:10 2018)\nconf_on_shared_storage=True\nmaintenance=True\nstate=LocalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 0, "stopped": false, "maintenance": true, "crc32": "8a558591", "local_conf_timestamp": 4203, "host-ts": 4202}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4196 (Thu Sep 6 01:45:03 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4196 (Thu Sep 6 01:45:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a9977fba", "local_conf_timestamp": 4196, "host-ts": 4196}, "global_maintenance": false} root: INFO: * Engine has migrated. root: INFO: * Waiting For System Stability... cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 125, in local_maintenance _wait_for_engine_migration(host, he_index, "bad", "Migration Destination") File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 45, in _wait_for_engine_migration time.sleep(wait_value) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nlago.ssh: DEBUG: start task:c3e22bc1-2785-476e-8ba3-ad975b3823a0:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: end task:c3e22bc1-2785-476e-8ba3-ad975b3823a0:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: Running 4cc65a9e on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 4cc65a9e on lago-he-node-ng-suite-4-2-host-1 returned with 0\nlago.ssh: DEBUG: Command 4cc65a9e on lago-he-node-ng-suite-4-2-host-1 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4109 (Thu Sep 6 01:43:37 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=4109 (Thu Sep 6 01:43:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "044228a8", "local_conf_timestamp": 4109, "host-ts": 4109}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4115 (Thu Sep 6 01:43:42 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4115 (Thu Sep 6 01:43:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8d059996", "local_conf_timestamp": 4115, "host-ts": 4115}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:7f537510-6e18-4173-b0f5-d5f6a1c6e460:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:7f537510-6e18-4173-b0f5-d5f6a1c6e460:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 4dde7bbe on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local\nlago.ssh: DEBUG: Command 4dde7bbe on lago-he-node-ng-suite-4-2-host-0 returned with 0\nroot: INFO: * Waiting for engine to migrate...\nlago.ssh: DEBUG: start task:94120632-275d-478f-9cac-0ff1eb6c2e63:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:94120632-275d-478f-9cac-0ff1eb6c2e63:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 4ef81abe on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 4ef81abe on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 4ef81abe on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4119 (Thu Sep 6 01:43:47 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=4120 (Thu Sep 6 01:43:47 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8ad483f0", "local_conf_timestamp": 4120, "host-ts": 4119}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4115 (Thu Sep 6 01:43:42 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4115 (Thu Sep 6 01:43:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8d059996", "local_conf_timestamp": 4115, "host-ts": 4115}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:82d6fe01-3354-4477-9210-df6ac2d4d420:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:82d6fe01-3354-4477-9210-df6ac2d4d420:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 5626eb30 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 5626eb30 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 5626eb30 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4131 (Thu Sep 6 01:43:58 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4130 (Thu Sep 6 01:43:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "65ca1ba5", "local_conf_timestamp": 4130, "host-ts": 4131}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4125 (Thu Sep 6 01:43:53 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4126 (Thu Sep 6 01:43:53 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "18c7c923", "local_conf_timestamp": 4126, "host-ts": 4125}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:b3211ae7-4af2-469b-92c0-38ab6f07c5c4:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:b3211ae7-4af2-469b-92c0-38ab6f07c5c4:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 59979f1c on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 59979f1c on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 59979f1c on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4141 (Thu Sep 6 01:44:09 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4141 (Thu Sep 6 01:44:09 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "49290bc0", "local_conf_timestamp": 4141, "host-ts": 4141}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4135 (Thu Sep 6 01:44:02 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4135 (Thu Sep 6 01:44:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8588679c", "local_conf_timestamp": 4135, "host-ts": 4135}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:f62305f7-c324-4ee2-ba11-d092ac823600:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:f62305f7-c324-4ee2-ba11-d092ac823600:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 5dfd4692 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 5dfd4692 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 5dfd4692 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4151 (Thu Sep 6 01:44:18 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4151 (Thu Sep 6 01:44:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e71a3502", "local_conf_timestamp": 4151, "host-ts": 4151}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4145 (Thu Sep 6 01:44:13 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4145 (Thu Sep 6 01:44:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ef65af1f", "local_conf_timestamp": 4145, "host-ts": 4145}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:a553c343-1311-45f3-9fc3-ed53d93b21d5:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:a553c343-1311-45f3-9fc3-ed53d93b21d5:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 61aded00 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 61aded00 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 61aded00 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4151 (Thu Sep 6 01:44:18 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4151 (Thu Sep 6 01:44:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e71a3502", "local_conf_timestamp": 4151, "host-ts": 4151}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4145 (Thu Sep 6 01:44:13 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4145 (Thu Sep 6 01:44:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ef65af1f", "local_conf_timestamp": 4145, "host-ts": 4145}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:bb63e900-36b1-47d5-8bd7-b6b67eed6f7e:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:bb63e900-36b1-47d5-8bd7-b6b67eed6f7e:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 64b1136a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 64b1136a on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 64b1136a on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4151 (Thu Sep 6 01:44:18 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4151 (Thu Sep 6 01:44:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e71a3502", "local_conf_timestamp": 4151, "host-ts": 4151}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4155 (Thu Sep 6 01:44:22 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4155 (Thu Sep 6 01:44:22 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eb23501a", "local_conf_timestamp": 4155, "host-ts": 4155}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:97447141-6b3c-4188-ad5b-a8bbe1cc45db:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:97447141-6b3c-4188-ad5b-a8bbe1cc45db:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 67a5dc18 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 67a5dc18 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 67a5dc18 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4161 (Thu Sep 6 01:44:29 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4162 (Thu Sep 6 01:44:30 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "573721bf", "local_conf_timestamp": 4162, "host-ts": 4161}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4155 (Thu Sep 6 01:44:22 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4155 (Thu Sep 6 01:44:22 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eb23501a", "local_conf_timestamp": 4155, "host-ts": 4155}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:dcbacc88-4c96-4575-a53a-31b073b1e31e:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:dcbacc88-4c96-4575-a53a-31b073b1e31e:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 6ada63f4 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 6ada63f4 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 6ada63f4 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4171 (Thu Sep 6 01:44:39 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4172 (Thu Sep 6 01:44:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f9041f7d", "local_conf_timestamp": 4172, "host-ts": 4171}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4165 (Thu Sep 6 01:44:32 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4165 (Thu Sep 6 01:44:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "54cc899b", "local_conf_timestamp": 4165, "host-ts": 4165}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:90045a52-c293-4837-bfef-33a5a1910dc5:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:90045a52-c293-4837-bfef-33a5a1910dc5:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 6df25e2a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 6df25e2a on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 6df25e2a on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4171 (Thu Sep 6 01:44:39 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4172 (Thu Sep 6 01:44:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f9041f7d", "local_conf_timestamp": 4172, "host-ts": 4171}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4165 (Thu Sep 6 01:44:32 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4165 (Thu Sep 6 01:44:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "54cc899b", "local_conf_timestamp": 4165, "host-ts": 4165}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:a7476663-b52b-4275-8cc2-cd9ff3b88a3c:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:a7476663-b52b-4275-8cc2-cd9ff3b88a3c:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 71077262 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 71077262 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 71077262 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4181 (Thu Sep 6 01:44:48 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4181 (Thu Sep 6 01:44:49 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "edb81fdc", "local_conf_timestamp": 4181, "host-ts": 4181}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4175 (Thu Sep 6 01:44:43 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4176 (Thu Sep 6 01:44:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "c9832724", "local_conf_timestamp": 4176, "host-ts": 4175}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:42a25e42-c90a-456e-8669-82df79c7507c:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:42a25e42-c90a-456e-8669-82df79c7507c:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 74595c8c on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 74595c8c on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 74595c8c on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4181 (Thu Sep 6 01:44:48 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4181 (Thu Sep 6 01:44:49 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "edb81fdc", "local_conf_timestamp": 4181, "host-ts": 4181}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4175 (Thu Sep 6 01:44:43 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4176 (Thu Sep 6 01:44:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "c9832724", "local_conf_timestamp": 4176, "host-ts": 4175}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:22a966cf-52a3-43b7-98b0-a28f0df8f67e:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:22a966cf-52a3-43b7-98b0-a28f0df8f67e:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 77bc4bdc on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 77bc4bdc on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 77bc4bdc on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4191 (Thu Sep 6 01:44:59 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4192 (Thu Sep 6 01:45:00 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "VM migrated away successfully", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "29ba0d11", "local_conf_timestamp": 4192, "host-ts": 4191}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4185 (Thu Sep 6 01:44:52 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4185 (Thu Sep 6 01:44:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "6f6d73ad", "local_conf_timestamp": 4185, "host-ts": 4185}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:f9fa6fdb-2c5a-41b9-ac58-6ec7a8e8ef3f:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:f9fa6fdb-2c5a-41b9-ac58-6ec7a8e8ef3f:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 78d063fa on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 78d063fa on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 78d063fa on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4191 (Thu Sep 6 01:44:59 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4192 (Thu Sep 6 01:45:00 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "VM migrated away successfully", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "29ba0d11", "local_conf_timestamp": 4192, "host-ts": 4191}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4185 (Thu Sep 6 01:44:52 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4185 (Thu Sep 6 01:44:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "6f6d73ad", "local_conf_timestamp": 4185, "host-ts": 4185}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:095cd3ad-d465-46bb-8cd1-fd07a1a124ed:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:095cd3ad-d465-46bb-8cd1-fd07a1a124ed:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 7fdb3a94 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 7fdb3a94 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 7fdb3a94 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4202 (Thu Sep 6 01:45:10 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=4203 (Thu Sep 6 01:45:10 2018)\\nconf_on_shared_storage=True\\nmaintenance=True\\nstate=LocalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 0, "stopped": false, "maintenance": true, "crc32": "8a558591", "local_conf_timestamp": 4203, "host-ts": 4202}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4196 (Thu Sep 6 01:45:03 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4196 (Thu Sep 6 01:45:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a9977fba", "local_conf_timestamp": 4196, "host-ts": 4196}, "global_maintenance": false}\n\nroot: INFO: * Engine has migrated.\nroot: INFO: * Waiting For System Stability...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/286/ Build Number: 286 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #285 [Gal Ben Haim] Adding dr suite [Barak Korren] Remove the populate_mock function from mock_runner [Daniel Belenky] mock_runner: store shell cmd in a variable [Barak Korren] mock_runner: Added timeout param [Barak Korren] Make whitelist repo configurable via env vars [Daniel Belenky] stdci_runner: let mock_runner manage timeout [Greg Sheremeta] remove 4.1 change queue from ovirt-engine-nodejs config Changes for Build #286 [Gal Ben Haim] Adding dr suite ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 010_local_maintenance_cli.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... lago.ssh: DEBUG: start task:73fa5f94-c894-462a-a6ef-b9ebc7cc0119:Get ssh client for lago-he-node-ng-suite-4-2-host-1: lago.ssh: DEBUG: end task:73fa5f94-c894-462a-a6ef-b9ebc7cc0119:Get ssh client for lago-he-node-ng-suite-4-2-host-1: lago.ssh: DEBUG: Running 8e620cda on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 8e620cda on lago-he-node-ng-suite-4-2-host-1 returned with 0 lago.ssh: DEBUG: Command 8e620cda on lago-he-node-ng-suite-4-2-host-1 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4150 (Thu Sep 6 09:45:11 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4150 (Thu Sep 6 09:45:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "242f0670", "local_conf_timestamp": 4150, "host-ts": 4150}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4152 (Thu Sep 6 09:45:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4153 (Thu Sep 6 09:45:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8cf42a04", "local_conf_timestamp": 4153, "host-ts": 4152}, "global_maintenance": false} lago.ssh: DEBUG: start task:90e0be1f-91dc-4061-ae75-7f6499a7d220:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:90e0be1f-91dc-4061-ae75-7f6499a7d220:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 8f7dc4e2 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local lago.ssh: DEBUG: Command 8f7dc4e2 on lago-he-node-ng-suite-4-2-host-0 returned with 0 root: INFO: * Waiting for engine to migrate... lago.ssh: DEBUG: start task:009baaf8-0bcc-416c-a40b-5158836fdd91:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:009baaf8-0bcc-416c-a40b-5158836fdd91:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 9090d7c0 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9090d7c0 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 9090d7c0 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4150 (Thu Sep 6 09:45:11 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4150 (Thu Sep 6 09:45:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "242f0670", "local_conf_timestamp": 4150, "host-ts": 4150}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4152 (Thu Sep 6 09:45:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4153 (Thu Sep 6 09:45:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8cf42a04", "local_conf_timestamp": 4153, "host-ts": 4152}, "global_maintenance": false} lago.ssh: DEBUG: start task:be4b28ac-98d6-4331-b730-ad8226e99068:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:be4b28ac-98d6-4331-b730-ad8226e99068:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 98502268 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 98502268 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 98502268 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4161 (Thu Sep 6 09:45:22 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4160 (Thu Sep 6 09:45:22 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7ed257e4", "local_conf_timestamp": 4160, "host-ts": 4161}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4152 (Thu Sep 6 09:45:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4153 (Thu Sep 6 09:45:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8cf42a04", "local_conf_timestamp": 4153, "host-ts": 4152}, "global_maintenance": false} lago.ssh: DEBUG: start task:21a75a1e-6a19-44dd-9e03-2b93673c2cf7:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:21a75a1e-6a19-44dd-9e03-2b93673c2cf7:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 9b52ff1c on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9b52ff1c on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 9b52ff1c on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4171 (Thu Sep 6 09:45:33 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4171 (Thu Sep 6 09:45:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "600d4ec7", "local_conf_timestamp": 4171, "host-ts": 4171}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4163 (Thu Sep 6 09:45:24 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4163 (Thu Sep 6 09:45:25 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ddc3e83", "local_conf_timestamp": 4163, "host-ts": 4163}, "global_maintenance": false} cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 125, in local_maintenance _wait_for_engine_migration(host, he_index, "bad", "Migration Destination") File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 36, in _wait_for_engine_migration testlib.assert_true_within_long(lambda: _get_he_status(host) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 246, in assert_equals_within time.sleep(3) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nlago.ssh: DEBUG: start task:73fa5f94-c894-462a-a6ef-b9ebc7cc0119:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: end task:73fa5f94-c894-462a-a6ef-b9ebc7cc0119:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: Running 8e620cda on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 8e620cda on lago-he-node-ng-suite-4-2-host-1 returned with 0\nlago.ssh: DEBUG: Command 8e620cda on lago-he-node-ng-suite-4-2-host-1 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4150 (Thu Sep 6 09:45:11 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=4150 (Thu Sep 6 09:45:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "242f0670", "local_conf_timestamp": 4150, "host-ts": 4150}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4152 (Thu Sep 6 09:45:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4153 (Thu Sep 6 09:45:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8cf42a04", "local_conf_timestamp": 4153, "host-ts": 4152}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:90e0be1f-91dc-4061-ae75-7f6499a7d220:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:90e0be1f-91dc-4061-ae75-7f6499a7d220:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 8f7dc4e2 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local\nlago.ssh: DEBUG: Command 8f7dc4e2 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nroot: INFO: * Waiting for engine to migrate...\nlago.ssh: DEBUG: start task:009baaf8-0bcc-416c-a40b-5158836fdd91:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:009baaf8-0bcc-416c-a40b-5158836fdd91:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 9090d7c0 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9090d7c0 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 9090d7c0 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4150 (Thu Sep 6 09:45:11 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=4150 (Thu Sep 6 09:45:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "242f0670", "local_conf_timestamp": 4150, "host-ts": 4150}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4152 (Thu Sep 6 09:45:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4153 (Thu Sep 6 09:45:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8cf42a04", "local_conf_timestamp": 4153, "host-ts": 4152}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:be4b28ac-98d6-4331-b730-ad8226e99068:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:be4b28ac-98d6-4331-b730-ad8226e99068:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 98502268 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 98502268 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 98502268 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4161 (Thu Sep 6 09:45:22 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4160 (Thu Sep 6 09:45:22 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7ed257e4", "local_conf_timestamp": 4160, "host-ts": 4161}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4152 (Thu Sep 6 09:45:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4153 (Thu Sep 6 09:45:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8cf42a04", "local_conf_timestamp": 4153, "host-ts": 4152}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:21a75a1e-6a19-44dd-9e03-2b93673c2cf7:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:21a75a1e-6a19-44dd-9e03-2b93673c2cf7:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 9b52ff1c on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9b52ff1c on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 9b52ff1c on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4171 (Thu Sep 6 09:45:33 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4171 (Thu Sep 6 09:45:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "600d4ec7", "local_conf_timestamp": 4171, "host-ts": 4171}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4163 (Thu Sep 6 09:45:24 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4163 (Thu Sep 6 09:45:25 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ddc3e83", "local_conf_timestamp": 4163, "host-ts": 4163}, "global_maintenance": false}\n\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/287/ Build Number: 287 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #285 [Gal Ben Haim] Adding dr suite [Barak Korren] Remove the populate_mock function from mock_runner [Daniel Belenky] mock_runner: store shell cmd in a variable [Barak Korren] mock_runner: Added timeout param [Barak Korren] Make whitelist repo configurable via env vars [Daniel Belenky] stdci_runner: let mock_runner manage timeout [Greg Sheremeta] remove 4.1 change queue from ovirt-engine-nodejs config Changes for Build #286 [Gal Ben Haim] Adding dr suite Changes for Build #287 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file [Barak Korren] Enable CI for kubevirt/client-python [Daniel Belenky] Add timeout config to stdci dsl [Daniel Belenky] stdci_runner.groovy: utilize DSL's timeout cfg ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 010_local_maintenance_cli.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... lago.ssh: DEBUG: start task:dbac6aee-f807-447a-8c86-15134f2b7749:Get ssh client for lago-he-node-ng-suite-4-2-host-1: lago.ssh: DEBUG: end task:dbac6aee-f807-447a-8c86-15134f2b7749:Get ssh client for lago-he-node-ng-suite-4-2-host-1: lago.ssh: DEBUG: Running daa5b834 on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json lago.ssh: DEBUG: Command daa5b834 on lago-he-node-ng-suite-4-2-host-1 returned with 0 lago.ssh: DEBUG: Command daa5b834 on lago-he-node-ng-suite-4-2-host-1 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4146 (Thu Sep 6 17:39:36 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4146 (Thu Sep 6 17:39:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e9066a29", "local_conf_timestamp": 4146, "host-ts": 4146}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4153 (Thu Sep 6 17:39:43 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4153 (Thu Sep 6 17:39:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "662a0a14", "local_conf_timestamp": 4153, "host-ts": 4153}, "global_maintenance": false} lago.ssh: DEBUG: start task:d30f82be-086c-4823-8307-6871f22336f6:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:d30f82be-086c-4823-8307-6871f22336f6:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running dbb9947a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local lago.ssh: DEBUG: Command dbb9947a on lago-he-node-ng-suite-4-2-host-0 returned with 0 root: INFO: * Waiting for engine to migrate... lago.ssh: DEBUG: start task:7c4ae3ab-7501-4c92-9bea-fdaaef4859b2:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:7c4ae3ab-7501-4c92-9bea-fdaaef4859b2:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running dce9aff6 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command dce9aff6 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command dce9aff6 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4156 (Thu Sep 6 17:39:46 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4156 (Thu Sep 6 17:39:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "35612a16", "local_conf_timestamp": 4156, "host-ts": 4156}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4153 (Thu Sep 6 17:39:43 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4153 (Thu Sep 6 17:39:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "662a0a14", "local_conf_timestamp": 4153, "host-ts": 4153}, "global_maintenance": false} lago.ssh: DEBUG: start task:eb74d1b1-2d4e-416b-a20f-1bc0ce171837:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:eb74d1b1-2d4e-416b-a20f-1bc0ce171837:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running e4cbecf2 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e4cbecf2 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command e4cbecf2 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4177 (Thu Sep 6 17:40:07 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4177 (Thu Sep 6 17:40:08 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "40657674", "local_conf_timestamp": 4177, "host-ts": 4177}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4173 (Thu Sep 6 17:40:03 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4174 (Thu Sep 6 17:40:04 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f0c361bd", "local_conf_timestamp": 4174, "host-ts": 4173}, "global_maintenance": false} lago.ssh: DEBUG: start task:9b2117bb-3e9b-4784-bb67-0d5f98f9d7b5:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:9b2117bb-3e9b-4784-bb67-0d5f98f9d7b5:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running e8606ab4 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e8606ab4 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command e8606ab4 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4177 (Thu Sep 6 17:40:07 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4177 (Thu Sep 6 17:40:08 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "40657674", "local_conf_timestamp": 4177, "host-ts": 4177}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4173 (Thu Sep 6 17:40:03 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4174 (Thu Sep 6 17:40:04 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f0c361bd", "local_conf_timestamp": 4174, "host-ts": 4173}, "global_maintenance": false} lago.ssh: DEBUG: start task:ddbde6de-1c5d-42c4-9156-a1c592cb6dd0:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:ddbde6de-1c5d-42c4-9156-a1c592cb6dd0:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running eb997522 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command eb997522 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command eb997522 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4187 (Thu Sep 6 17:40:17 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4187 (Thu Sep 6 17:40:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "cdd0276f", "local_conf_timestamp": 4187, "host-ts": 4187}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4183 (Thu Sep 6 17:40:13 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4183 (Thu Sep 6 17:40:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "267dc4b5", "local_conf_timestamp": 4183, "host-ts": 4183}, "global_maintenance": false} lago.ssh: DEBUG: start task:fc0e0ad5-af11-42b1-8b1a-542db357c10f:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:fc0e0ad5-af11-42b1-8b1a-542db357c10f:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running ee95607e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ee95607e on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command ee95607e on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4187 (Thu Sep 6 17:40:17 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4187 (Thu Sep 6 17:40:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "cdd0276f", "local_conf_timestamp": 4187, "host-ts": 4187}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4183 (Thu Sep 6 17:40:13 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4183 (Thu Sep 6 17:40:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "267dc4b5", "local_conf_timestamp": 4183, "host-ts": 4183}, "global_maintenance": false} lago.ssh: DEBUG: start task:33c75a20-69a6-4f1f-9d10-5964b70e5476:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:33c75a20-69a6-4f1f-9d10-5964b70e5476:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running f22070d0 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f22070d0 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command f22070d0 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4197 (Thu Sep 6 17:40:27 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4197 (Thu Sep 6 17:40:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "63e319ad", "local_conf_timestamp": 4197, "host-ts": 4197}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4193 (Thu Sep 6 17:40:23 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4193 (Thu Sep 6 17:40:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "223b3bb0", "local_conf_timestamp": 4193, "host-ts": 4193}, "global_maintenance": false} lago.ssh: DEBUG: start task:eebbdb9b-3cca-497b-83fb-90f5b365bd00:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:eebbdb9b-3cca-497b-83fb-90f5b365bd00:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running f5324a00 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f5324a00 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command f5324a00 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4197 (Thu Sep 6 17:40:27 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4197 (Thu Sep 6 17:40:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "63e319ad", "local_conf_timestamp": 4197, "host-ts": 4197}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4193 (Thu Sep 6 17:40:23 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4193 (Thu Sep 6 17:40:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "223b3bb0", "local_conf_timestamp": 4193, "host-ts": 4193}, "global_maintenance": false} lago.ssh: DEBUG: start task:901e9f22-e329-4658-a951-0e1aa80b3cd6:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:901e9f22-e329-4658-a951-0e1aa80b3cd6:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running f81411ae on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f81411ae on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command f81411ae on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4207 (Thu Sep 6 17:40:37 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4207 (Thu Sep 6 17:40:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2253735e", "local_conf_timestamp": 4207, "host-ts": 4207}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4203 (Thu Sep 6 17:40:33 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4203 (Thu Sep 6 17:40:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "27dd3283", "local_conf_timestamp": 4203, "host-ts": 4203}, "global_maintenance": false} lago.ssh: DEBUG: start task:e3708b7a-92ad-418c-812b-62b5a7848558:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:e3708b7a-92ad-418c-812b-62b5a7848558:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running fb99b770 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command fb99b770 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command fb99b770 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4207 (Thu Sep 6 17:40:37 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4207 (Thu Sep 6 17:40:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2253735e", "local_conf_timestamp": 4207, "host-ts": 4207}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4203 (Thu Sep 6 17:40:33 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4203 (Thu Sep 6 17:40:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "27dd3283", "local_conf_timestamp": 4203, "host-ts": 4203}, "global_maintenance": false} lago.ssh: DEBUG: start task:ac8d327c-bc33-4c15-81c4-091c7578adfc:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:ac8d327c-bc33-4c15-81c4-091c7578adfc:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running fe9fa182 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command fe9fa182 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command fe9fa182 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4217 (Thu Sep 6 17:40:48 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4217 (Thu Sep 6 17:40:48 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "8c604d9c", "local_conf_timestamp": 4217, "host-ts": 4217}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4213 (Thu Sep 6 17:40:43 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4213 (Thu Sep 6 17:40:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "239bcd86", "local_conf_timestamp": 4213, "host-ts": 4213}, "global_maintenance": false} lago.ssh: DEBUG: start task:6b6e4c48-4da2-4561-8e20-81c510232a9b:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:6b6e4c48-4da2-4561-8e20-81c510232a9b:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 018219fc on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 018219fc on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 018219fc on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4217 (Thu Sep 6 17:40:48 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4217 (Thu Sep 6 17:40:48 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "8c604d9c", "local_conf_timestamp": 4217, "host-ts": 4217}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4213 (Thu Sep 6 17:40:43 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4213 (Thu Sep 6 17:40:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "239bcd86", "local_conf_timestamp": 4213, "host-ts": 4213}, "global_maintenance": false} lago.ssh: DEBUG: start task:96227485-3980-4fdc-8b7a-6b8209beddba:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:96227485-3980-4fdc-8b7a-6b8209beddba:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 0464194a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 0464194a on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 0464194a on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4227 (Thu Sep 6 17:40:57 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4227 (Thu Sep 6 17:40:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "c32ada44", "local_conf_timestamp": 4227, "host-ts": 4227}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4223 (Thu Sep 6 17:40:53 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4224 (Thu Sep 6 17:40:54 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "b134592a", "local_conf_timestamp": 4224, "host-ts": 4223}, "global_maintenance": false} lago.ssh: DEBUG: start task:ff150e2b-4824-4415-bb85-e03c385d8164:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:ff150e2b-4824-4415-bb85-e03c385d8164:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 05762a44 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 05762a44 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 05762a44 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4227 (Thu Sep 6 17:40:57 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4227 (Thu Sep 6 17:40:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "c32ada44", "local_conf_timestamp": 4227, "host-ts": 4227}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4223 (Thu Sep 6 17:40:53 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4224 (Thu Sep 6 17:40:54 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "b134592a", "local_conf_timestamp": 4224, "host-ts": 4223}, "global_maintenance": false} lago.ssh: DEBUG: start task:277690ad-d660-43ec-91d6-219e1f44d4a3:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:277690ad-d660-43ec-91d6-219e1f44d4a3:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 0c8f00d0 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 0c8f00d0 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 0c8f00d0 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4237 (Thu Sep 6 17:41:07 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=4237 (Thu Sep 6 17:41:08 2018)\nconf_on_shared_storage=True\nmaintenance=True\nstate=LocalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 0, "stopped": false, "maintenance": true, "crc32": "055173ed", "local_conf_timestamp": 4237, "host-ts": 4237}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4233 (Thu Sep 6 17:41:03 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4233 (Thu Sep 6 17:41:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "86f105ab", "local_conf_timestamp": 4233, "host-ts": 4233}, "global_maintenance": false} root: INFO: * Engine has migrated. root: INFO: * Waiting For System Stability... cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 125, in local_maintenance _wait_for_engine_migration(host, he_index, "bad", "Migration Destination") File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 45, in _wait_for_engine_migration time.sleep(wait_value) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nlago.ssh: DEBUG: start task:dbac6aee-f807-447a-8c86-15134f2b7749:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: end task:dbac6aee-f807-447a-8c86-15134f2b7749:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: Running daa5b834 on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command daa5b834 on lago-he-node-ng-suite-4-2-host-1 returned with 0\nlago.ssh: DEBUG: Command daa5b834 on lago-he-node-ng-suite-4-2-host-1 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4146 (Thu Sep 6 17:39:36 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=4146 (Thu Sep 6 17:39:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e9066a29", "local_conf_timestamp": 4146, "host-ts": 4146}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4153 (Thu Sep 6 17:39:43 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4153 (Thu Sep 6 17:39:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "662a0a14", "local_conf_timestamp": 4153, "host-ts": 4153}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:d30f82be-086c-4823-8307-6871f22336f6:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:d30f82be-086c-4823-8307-6871f22336f6:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running dbb9947a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local\nlago.ssh: DEBUG: Command dbb9947a on lago-he-node-ng-suite-4-2-host-0 returned with 0\nroot: INFO: * Waiting for engine to migrate...\nlago.ssh: DEBUG: start task:7c4ae3ab-7501-4c92-9bea-fdaaef4859b2:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:7c4ae3ab-7501-4c92-9bea-fdaaef4859b2:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running dce9aff6 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command dce9aff6 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command dce9aff6 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4156 (Thu Sep 6 17:39:46 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=4156 (Thu Sep 6 17:39:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "35612a16", "local_conf_timestamp": 4156, "host-ts": 4156}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4153 (Thu Sep 6 17:39:43 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4153 (Thu Sep 6 17:39:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "662a0a14", "local_conf_timestamp": 4153, "host-ts": 4153}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:eb74d1b1-2d4e-416b-a20f-1bc0ce171837:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:eb74d1b1-2d4e-416b-a20f-1bc0ce171837:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running e4cbecf2 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e4cbecf2 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command e4cbecf2 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4177 (Thu Sep 6 17:40:07 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4177 (Thu Sep 6 17:40:08 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "40657674", "local_conf_timestamp": 4177, "host-ts": 4177}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4173 (Thu Sep 6 17:40:03 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4174 (Thu Sep 6 17:40:04 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f0c361bd", "local_conf_timestamp": 4174, "host-ts": 4173}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:9b2117bb-3e9b-4784-bb67-0d5f98f9d7b5:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:9b2117bb-3e9b-4784-bb67-0d5f98f9d7b5:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running e8606ab4 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e8606ab4 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command e8606ab4 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4177 (Thu Sep 6 17:40:07 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4177 (Thu Sep 6 17:40:08 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "40657674", "local_conf_timestamp": 4177, "host-ts": 4177}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4173 (Thu Sep 6 17:40:03 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4174 (Thu Sep 6 17:40:04 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f0c361bd", "local_conf_timestamp": 4174, "host-ts": 4173}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:ddbde6de-1c5d-42c4-9156-a1c592cb6dd0:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:ddbde6de-1c5d-42c4-9156-a1c592cb6dd0:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running eb997522 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command eb997522 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command eb997522 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4187 (Thu Sep 6 17:40:17 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4187 (Thu Sep 6 17:40:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "cdd0276f", "local_conf_timestamp": 4187, "host-ts": 4187}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4183 (Thu Sep 6 17:40:13 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4183 (Thu Sep 6 17:40:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "267dc4b5", "local_conf_timestamp": 4183, "host-ts": 4183}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:fc0e0ad5-af11-42b1-8b1a-542db357c10f:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:fc0e0ad5-af11-42b1-8b1a-542db357c10f:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running ee95607e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ee95607e on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command ee95607e on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4187 (Thu Sep 6 17:40:17 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4187 (Thu Sep 6 17:40:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "cdd0276f", "local_conf_timestamp": 4187, "host-ts": 4187}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4183 (Thu Sep 6 17:40:13 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4183 (Thu Sep 6 17:40:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "267dc4b5", "local_conf_timestamp": 4183, "host-ts": 4183}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:33c75a20-69a6-4f1f-9d10-5964b70e5476:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:33c75a20-69a6-4f1f-9d10-5964b70e5476:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running f22070d0 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f22070d0 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command f22070d0 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4197 (Thu Sep 6 17:40:27 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4197 (Thu Sep 6 17:40:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "63e319ad", "local_conf_timestamp": 4197, "host-ts": 4197}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4193 (Thu Sep 6 17:40:23 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4193 (Thu Sep 6 17:40:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "223b3bb0", "local_conf_timestamp": 4193, "host-ts": 4193}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:eebbdb9b-3cca-497b-83fb-90f5b365bd00:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:eebbdb9b-3cca-497b-83fb-90f5b365bd00:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running f5324a00 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f5324a00 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command f5324a00 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4197 (Thu Sep 6 17:40:27 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4197 (Thu Sep 6 17:40:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "63e319ad", "local_conf_timestamp": 4197, "host-ts": 4197}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4193 (Thu Sep 6 17:40:23 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4193 (Thu Sep 6 17:40:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "223b3bb0", "local_conf_timestamp": 4193, "host-ts": 4193}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:901e9f22-e329-4658-a951-0e1aa80b3cd6:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:901e9f22-e329-4658-a951-0e1aa80b3cd6:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running f81411ae on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f81411ae on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command f81411ae on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4207 (Thu Sep 6 17:40:37 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4207 (Thu Sep 6 17:40:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2253735e", "local_conf_timestamp": 4207, "host-ts": 4207}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4203 (Thu Sep 6 17:40:33 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4203 (Thu Sep 6 17:40:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "27dd3283", "local_conf_timestamp": 4203, "host-ts": 4203}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:e3708b7a-92ad-418c-812b-62b5a7848558:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:e3708b7a-92ad-418c-812b-62b5a7848558:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running fb99b770 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fb99b770 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command fb99b770 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4207 (Thu Sep 6 17:40:37 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4207 (Thu Sep 6 17:40:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2253735e", "local_conf_timestamp": 4207, "host-ts": 4207}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4203 (Thu Sep 6 17:40:33 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4203 (Thu Sep 6 17:40:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "27dd3283", "local_conf_timestamp": 4203, "host-ts": 4203}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:ac8d327c-bc33-4c15-81c4-091c7578adfc:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:ac8d327c-bc33-4c15-81c4-091c7578adfc:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running fe9fa182 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fe9fa182 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command fe9fa182 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4217 (Thu Sep 6 17:40:48 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4217 (Thu Sep 6 17:40:48 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "8c604d9c", "local_conf_timestamp": 4217, "host-ts": 4217}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4213 (Thu Sep 6 17:40:43 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4213 (Thu Sep 6 17:40:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "239bcd86", "local_conf_timestamp": 4213, "host-ts": 4213}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:6b6e4c48-4da2-4561-8e20-81c510232a9b:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:6b6e4c48-4da2-4561-8e20-81c510232a9b:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 018219fc on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 018219fc on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 018219fc on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4217 (Thu Sep 6 17:40:48 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4217 (Thu Sep 6 17:40:48 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "8c604d9c", "local_conf_timestamp": 4217, "host-ts": 4217}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4213 (Thu Sep 6 17:40:43 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4213 (Thu Sep 6 17:40:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "239bcd86", "local_conf_timestamp": 4213, "host-ts": 4213}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:96227485-3980-4fdc-8b7a-6b8209beddba:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:96227485-3980-4fdc-8b7a-6b8209beddba:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 0464194a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 0464194a on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 0464194a on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4227 (Thu Sep 6 17:40:57 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4227 (Thu Sep 6 17:40:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "c32ada44", "local_conf_timestamp": 4227, "host-ts": 4227}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4223 (Thu Sep 6 17:40:53 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4224 (Thu Sep 6 17:40:54 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "b134592a", "local_conf_timestamp": 4224, "host-ts": 4223}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:ff150e2b-4824-4415-bb85-e03c385d8164:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:ff150e2b-4824-4415-bb85-e03c385d8164:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 05762a44 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 05762a44 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 05762a44 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4227 (Thu Sep 6 17:40:57 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4227 (Thu Sep 6 17:40:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "c32ada44", "local_conf_timestamp": 4227, "host-ts": 4227}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4223 (Thu Sep 6 17:40:53 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4224 (Thu Sep 6 17:40:54 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "b134592a", "local_conf_timestamp": 4224, "host-ts": 4223}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:277690ad-d660-43ec-91d6-219e1f44d4a3:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:277690ad-d660-43ec-91d6-219e1f44d4a3:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 0c8f00d0 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 0c8f00d0 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 0c8f00d0 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4237 (Thu Sep 6 17:41:07 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=4237 (Thu Sep 6 17:41:08 2018)\\nconf_on_shared_storage=True\\nmaintenance=True\\nstate=LocalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 0, "stopped": false, "maintenance": true, "crc32": "055173ed", "local_conf_timestamp": 4237, "host-ts": 4237}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4233 (Thu Sep 6 17:41:03 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4233 (Thu Sep 6 17:41:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "86f105ab", "local_conf_timestamp": 4233, "host-ts": 4233}, "global_maintenance": false}\n\nroot: INFO: * Engine has migrated.\nroot: INFO: * Waiting For System Stability...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/288/ Build Number: 288 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #285 [Gal Ben Haim] Adding dr suite [Barak Korren] Remove the populate_mock function from mock_runner [Daniel Belenky] mock_runner: store shell cmd in a variable [Barak Korren] mock_runner: Added timeout param [Barak Korren] Make whitelist repo configurable via env vars [Daniel Belenky] stdci_runner: let mock_runner manage timeout [Greg Sheremeta] remove 4.1 change queue from ovirt-engine-nodejs config Changes for Build #286 [Gal Ben Haim] Adding dr suite Changes for Build #287 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file [Barak Korren] Enable CI for kubevirt/client-python [Daniel Belenky] Add timeout config to stdci dsl [Daniel Belenky] stdci_runner.groovy: utilize DSL's timeout cfg Changes for Build #288 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 010_local_maintenance_cli.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... lago.ssh: DEBUG: start task:c189ef82-a155-4e3d-89e3-fab0395faff6:Get ssh client for lago-he-node-ng-suite-4-2-host-1: lago.ssh: DEBUG: end task:c189ef82-a155-4e3d-89e3-fab0395faff6:Get ssh client for lago-he-node-ng-suite-4-2-host-1: lago.ssh: DEBUG: Running 0f8a5364 on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 0f8a5364 on lago-he-node-ng-suite-4-2-host-1 returned with 0 lago.ssh: DEBUG: Command 0f8a5364 on lago-he-node-ng-suite-4-2-host-1 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4189 (Fri Sep 7 01:40:47 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4189 (Fri Sep 7 01:40:47 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "51cd3615", "local_conf_timestamp": 4189, "host-ts": 4189}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4189 (Fri Sep 7 01:40:47 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4190 (Fri Sep 7 01:40:47 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e4aace33", "local_conf_timestamp": 4190, "host-ts": 4189}, "global_maintenance": false} lago.ssh: DEBUG: start task:26a5f00b-d6bc-425e-b389-be2240ae2a62:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:26a5f00b-d6bc-425e-b389-be2240ae2a62:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 1095a34e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local lago.ssh: DEBUG: Command 1095a34e on lago-he-node-ng-suite-4-2-host-0 returned with 0 root: INFO: * Waiting for engine to migrate... lago.ssh: DEBUG: start task:48184330-dd31-4722-bd98-47d467d9a50a:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:48184330-dd31-4722-bd98-47d467d9a50a:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 11c06ede on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 11c06ede on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 11c06ede on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4199 (Fri Sep 7 01:40:57 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4200 (Fri Sep 7 01:40:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "27f2cb9e", "local_conf_timestamp": 4200, "host-ts": 4199}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4199 (Fri Sep 7 01:40:56 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4199 (Fri Sep 7 01:40:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "802bb8d3", "local_conf_timestamp": 4199, "host-ts": 4199}, "global_maintenance": false} lago.ssh: DEBUG: start task:145cadee-7aef-45ac-8890-4ce461b411f1:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:145cadee-7aef-45ac-8890-4ce461b411f1:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 195ce73a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 195ce73a on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 195ce73a on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4209 (Fri Sep 7 01:41:07 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4209 (Fri Sep 7 01:41:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2792a742", "local_conf_timestamp": 4209, "host-ts": 4209}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4209 (Fri Sep 7 01:41:06 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4209 (Fri Sep 7 01:41:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "36e9696e", "local_conf_timestamp": 4209, "host-ts": 4209}, "global_maintenance": false} lago.ssh: DEBUG: start task:703d9155-8ba7-437d-a93b-936eb2a108cf:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:703d9155-8ba7-437d-a93b-936eb2a108cf:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 1cc64574 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 1cc64574 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 1cc64574 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4219 (Fri Sep 7 01:41:16 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4219 (Fri Sep 7 01:41:16 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d3682b6b", "local_conf_timestamp": 4219, "host-ts": 4219}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4219 (Fri Sep 7 01:41:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4219 (Fri Sep 7 01:41:16 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "4592a3f3", "local_conf_timestamp": 4219, "host-ts": 4219}, "global_maintenance": false} lago.ssh: DEBUG: start task:6e1755c1-39f7-4af1-8be7-ce70f15fdc71:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:6e1755c1-39f7-4af1-8be7-ce70f15fdc71:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 2007c334 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 2007c334 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 2007c334 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4219 (Fri Sep 7 01:41:16 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4219 (Fri Sep 7 01:41:16 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d3682b6b", "local_conf_timestamp": 4219, "host-ts": 4219}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4219 (Fri Sep 7 01:41:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4219 (Fri Sep 7 01:41:16 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "4592a3f3", "local_conf_timestamp": 4219, "host-ts": 4219}, "global_maintenance": false} lago.ssh: DEBUG: start task:0af48f5c-3fac-4ef9-bc54-6bac7372e1dc:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:0af48f5c-3fac-4ef9-bc54-6bac7372e1dc:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 22f4fc10 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 22f4fc10 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 22f4fc10 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4229 (Fri Sep 7 01:41:27 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4229 (Fri Sep 7 01:41:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "fa4c6e6c", "local_conf_timestamp": 4229, "host-ts": 4229}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4229 (Fri Sep 7 01:41:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4229 (Fri Sep 7 01:41:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "4959a2fc", "local_conf_timestamp": 4229, "host-ts": 4229}, "global_maintenance": false} lago.ssh: DEBUG: start task:6c5b7e87-9726-4d52-a513-d5612c080648:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:6c5b7e87-9726-4d52-a513-d5612c080648:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 2668ed7a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 2668ed7a on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 2668ed7a on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4229 (Fri Sep 7 01:41:27 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4229 (Fri Sep 7 01:41:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "fa4c6e6c", "local_conf_timestamp": 4229, "host-ts": 4229}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4229 (Fri Sep 7 01:41:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4229 (Fri Sep 7 01:41:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "4959a2fc", "local_conf_timestamp": 4229, "host-ts": 4229}, "global_maintenance": false} lago.ssh: DEBUG: start task:c5a1a83f-64a0-4f6d-b1f3-643e38d558a0:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:c5a1a83f-64a0-4f6d-b1f3-643e38d558a0:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 29c14fd0 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 29c14fd0 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 29c14fd0 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4239 (Fri Sep 7 01:41:36 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4239 (Fri Sep 7 01:41:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "547f50ae", "local_conf_timestamp": 4239, "host-ts": 4239}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4229 (Fri Sep 7 01:41:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4229 (Fri Sep 7 01:41:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "4959a2fc", "local_conf_timestamp": 4229, "host-ts": 4229}, "global_maintenance": false} lago.ssh: DEBUG: start task:00ff66e4-ea61-4b61-8e47-59f9acd927df:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:00ff66e4-ea61-4b61-8e47-59f9acd927df:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 2cb436ee on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 2cb436ee on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 2cb436ee on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4239 (Fri Sep 7 01:41:36 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4239 (Fri Sep 7 01:41:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "547f50ae", "local_conf_timestamp": 4239, "host-ts": 4239}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4229 (Fri Sep 7 01:41:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4229 (Fri Sep 7 01:41:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "4959a2fc", "local_conf_timestamp": 4229, "host-ts": 4229}, "global_maintenance": false} lago.ssh: DEBUG: start task:db7282fb-411f-4bf2-99e9-dd62f9045959:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:db7282fb-411f-4bf2-99e9-dd62f9045959:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 2fbda56e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 2fbda56e on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 2fbda56e on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4249 (Fri Sep 7 01:41:47 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4249 (Fri Sep 7 01:41:47 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "a804e462", "local_conf_timestamp": 4249, "host-ts": 4249}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4249 (Fri Sep 7 01:41:46 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4249 (Fri Sep 7 01:41:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e3eb786c", "local_conf_timestamp": 4249, "host-ts": 4249}, "global_maintenance": false} lago.ssh: DEBUG: start task:7e1be65f-4b45-4ffc-8ee4-95e630967e56:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:7e1be65f-4b45-4ffc-8ee4-95e630967e56:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 32c8a812 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 32c8a812 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 32c8a812 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4249 (Fri Sep 7 01:41:47 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4249 (Fri Sep 7 01:41:47 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "a804e462", "local_conf_timestamp": 4249, "host-ts": 4249}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4249 (Fri Sep 7 01:41:46 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4249 (Fri Sep 7 01:41:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e3eb786c", "local_conf_timestamp": 4249, "host-ts": 4249}, "global_maintenance": false} lago.ssh: DEBUG: start task:1b68579b-9706-4efb-9d2e-d320d3aa4529:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:1b68579b-9706-4efb-9d2e-d320d3aa4529:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 361e2104 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 361e2104 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 361e2104 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4259 (Fri Sep 7 01:41:56 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4259 (Fri Sep 7 01:41:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "0637daa0", "local_conf_timestamp": 4259, "host-ts": 4259}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4249 (Fri Sep 7 01:41:46 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4249 (Fri Sep 7 01:41:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e3eb786c", "local_conf_timestamp": 4249, "host-ts": 4249}, "global_maintenance": false} lago.ssh: DEBUG: start task:5ad0ec90-37c8-4156-9835-f6db4490d54f:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:5ad0ec90-37c8-4156-9835-f6db4490d54f:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 3967a538 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 3967a538 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 3967a538 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4259 (Fri Sep 7 01:41:56 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4259 (Fri Sep 7 01:41:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "0637daa0", "local_conf_timestamp": 4259, "host-ts": 4259}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4269 (Fri Sep 7 01:42:06 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4269 (Fri Sep 7 01:42:06 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "eb668666", "local_conf_timestamp": 4269, "host-ts": 4269}, "global_maintenance": false} lago.ssh: DEBUG: start task:c677703e-de42-499e-946b-6bb80d527e9b:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:c677703e-de42-499e-946b-6bb80d527e9b:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 3d2401bc on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 3d2401bc on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 3d2401bc on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4269 (Fri Sep 7 01:42:07 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4269 (Fri Sep 7 01:42:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2f139fa7", "local_conf_timestamp": 4269, "host-ts": 4269}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4269 (Fri Sep 7 01:42:06 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4269 (Fri Sep 7 01:42:06 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "eb668666", "local_conf_timestamp": 4269, "host-ts": 4269}, "global_maintenance": false} lago.ssh: DEBUG: start task:5fd905d8-2a2b-46d3-a93a-9d1ce753aa2e:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:5fd905d8-2a2b-46d3-a93a-9d1ce753aa2e:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 400293c6 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 400293c6 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 400293c6 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4269 (Fri Sep 7 01:42:07 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4269 (Fri Sep 7 01:42:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2f139fa7", "local_conf_timestamp": 4269, "host-ts": 4269}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4269 (Fri Sep 7 01:42:06 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4269 (Fri Sep 7 01:42:06 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "eb668666", "local_conf_timestamp": 4269, "host-ts": 4269}, "global_maintenance": false} lago.ssh: DEBUG: start task:435cf0a2-abce-4456-8259-1c54be3322bf:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:435cf0a2-abce-4456-8259-1c54be3322bf:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 42e032ba on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 42e032ba on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 42e032ba on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4279 (Fri Sep 7 01:42:16 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4279 (Fri Sep 7 01:42:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "aa89fffb", "local_conf_timestamp": 4279, "host-ts": 4279}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4279 (Fri Sep 7 01:42:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4279 (Fri Sep 7 01:42:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "725c7d7f", "local_conf_timestamp": 4279, "host-ts": 4279}, "global_maintenance": false} lago.ssh: DEBUG: start task:d712d611-7633-442e-ba76-7b8a7b151650:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:d712d611-7633-442e-ba76-7b8a7b151650:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 43f4ef60 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 43f4ef60 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 43f4ef60 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4279 (Fri Sep 7 01:42:16 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4279 (Fri Sep 7 01:42:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "aa89fffb", "local_conf_timestamp": 4279, "host-ts": 4279}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4279 (Fri Sep 7 01:42:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4279 (Fri Sep 7 01:42:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "725c7d7f", "local_conf_timestamp": 4279, "host-ts": 4279}, "global_maintenance": false} root: INFO: * Engine has migrated. root: INFO: * Waiting For System Stability... cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 125, in local_maintenance _wait_for_engine_migration(host, he_index, "bad", "Migration Destination") File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 45, in _wait_for_engine_migration time.sleep(wait_value) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nlago.ssh: DEBUG: start task:c189ef82-a155-4e3d-89e3-fab0395faff6:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: end task:c189ef82-a155-4e3d-89e3-fab0395faff6:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: Running 0f8a5364 on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 0f8a5364 on lago-he-node-ng-suite-4-2-host-1 returned with 0\nlago.ssh: DEBUG: Command 0f8a5364 on lago-he-node-ng-suite-4-2-host-1 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4189 (Fri Sep 7 01:40:47 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=4189 (Fri Sep 7 01:40:47 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "51cd3615", "local_conf_timestamp": 4189, "host-ts": 4189}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4189 (Fri Sep 7 01:40:47 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4190 (Fri Sep 7 01:40:47 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e4aace33", "local_conf_timestamp": 4190, "host-ts": 4189}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:26a5f00b-d6bc-425e-b389-be2240ae2a62:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:26a5f00b-d6bc-425e-b389-be2240ae2a62:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 1095a34e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local\nlago.ssh: DEBUG: Command 1095a34e on lago-he-node-ng-suite-4-2-host-0 returned with 0\nroot: INFO: * Waiting for engine to migrate...\nlago.ssh: DEBUG: start task:48184330-dd31-4722-bd98-47d467d9a50a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:48184330-dd31-4722-bd98-47d467d9a50a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 11c06ede on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 11c06ede on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 11c06ede on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4199 (Fri Sep 7 01:40:57 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=4200 (Fri Sep 7 01:40:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "27f2cb9e", "local_conf_timestamp": 4200, "host-ts": 4199}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4199 (Fri Sep 7 01:40:56 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4199 (Fri Sep 7 01:40:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "802bb8d3", "local_conf_timestamp": 4199, "host-ts": 4199}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:145cadee-7aef-45ac-8890-4ce461b411f1:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:145cadee-7aef-45ac-8890-4ce461b411f1:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 195ce73a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 195ce73a on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 195ce73a on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4209 (Fri Sep 7 01:41:07 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4209 (Fri Sep 7 01:41:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2792a742", "local_conf_timestamp": 4209, "host-ts": 4209}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4209 (Fri Sep 7 01:41:06 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4209 (Fri Sep 7 01:41:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "36e9696e", "local_conf_timestamp": 4209, "host-ts": 4209}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:703d9155-8ba7-437d-a93b-936eb2a108cf:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:703d9155-8ba7-437d-a93b-936eb2a108cf:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 1cc64574 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 1cc64574 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 1cc64574 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4219 (Fri Sep 7 01:41:16 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4219 (Fri Sep 7 01:41:16 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d3682b6b", "local_conf_timestamp": 4219, "host-ts": 4219}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4219 (Fri Sep 7 01:41:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4219 (Fri Sep 7 01:41:16 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "4592a3f3", "local_conf_timestamp": 4219, "host-ts": 4219}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:6e1755c1-39f7-4af1-8be7-ce70f15fdc71:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:6e1755c1-39f7-4af1-8be7-ce70f15fdc71:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 2007c334 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 2007c334 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 2007c334 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4219 (Fri Sep 7 01:41:16 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4219 (Fri Sep 7 01:41:16 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d3682b6b", "local_conf_timestamp": 4219, "host-ts": 4219}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4219 (Fri Sep 7 01:41:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4219 (Fri Sep 7 01:41:16 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "4592a3f3", "local_conf_timestamp": 4219, "host-ts": 4219}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:0af48f5c-3fac-4ef9-bc54-6bac7372e1dc:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:0af48f5c-3fac-4ef9-bc54-6bac7372e1dc:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 22f4fc10 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 22f4fc10 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 22f4fc10 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4229 (Fri Sep 7 01:41:27 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4229 (Fri Sep 7 01:41:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "fa4c6e6c", "local_conf_timestamp": 4229, "host-ts": 4229}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4229 (Fri Sep 7 01:41:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4229 (Fri Sep 7 01:41:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "4959a2fc", "local_conf_timestamp": 4229, "host-ts": 4229}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:6c5b7e87-9726-4d52-a513-d5612c080648:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:6c5b7e87-9726-4d52-a513-d5612c080648:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 2668ed7a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 2668ed7a on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 2668ed7a on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4229 (Fri Sep 7 01:41:27 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4229 (Fri Sep 7 01:41:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "fa4c6e6c", "local_conf_timestamp": 4229, "host-ts": 4229}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4229 (Fri Sep 7 01:41:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4229 (Fri Sep 7 01:41:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "4959a2fc", "local_conf_timestamp": 4229, "host-ts": 4229}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:c5a1a83f-64a0-4f6d-b1f3-643e38d558a0:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:c5a1a83f-64a0-4f6d-b1f3-643e38d558a0:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 29c14fd0 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 29c14fd0 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 29c14fd0 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4239 (Fri Sep 7 01:41:36 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4239 (Fri Sep 7 01:41:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "547f50ae", "local_conf_timestamp": 4239, "host-ts": 4239}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4229 (Fri Sep 7 01:41:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4229 (Fri Sep 7 01:41:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "4959a2fc", "local_conf_timestamp": 4229, "host-ts": 4229}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:00ff66e4-ea61-4b61-8e47-59f9acd927df:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:00ff66e4-ea61-4b61-8e47-59f9acd927df:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 2cb436ee on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 2cb436ee on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 2cb436ee on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4239 (Fri Sep 7 01:41:36 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4239 (Fri Sep 7 01:41:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "547f50ae", "local_conf_timestamp": 4239, "host-ts": 4239}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4229 (Fri Sep 7 01:41:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4229 (Fri Sep 7 01:41:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "4959a2fc", "local_conf_timestamp": 4229, "host-ts": 4229}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:db7282fb-411f-4bf2-99e9-dd62f9045959:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:db7282fb-411f-4bf2-99e9-dd62f9045959:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 2fbda56e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 2fbda56e on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 2fbda56e on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4249 (Fri Sep 7 01:41:47 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4249 (Fri Sep 7 01:41:47 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "a804e462", "local_conf_timestamp": 4249, "host-ts": 4249}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4249 (Fri Sep 7 01:41:46 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4249 (Fri Sep 7 01:41:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e3eb786c", "local_conf_timestamp": 4249, "host-ts": 4249}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:7e1be65f-4b45-4ffc-8ee4-95e630967e56:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:7e1be65f-4b45-4ffc-8ee4-95e630967e56:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 32c8a812 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 32c8a812 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 32c8a812 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4249 (Fri Sep 7 01:41:47 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4249 (Fri Sep 7 01:41:47 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "a804e462", "local_conf_timestamp": 4249, "host-ts": 4249}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4249 (Fri Sep 7 01:41:46 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4249 (Fri Sep 7 01:41:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e3eb786c", "local_conf_timestamp": 4249, "host-ts": 4249}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:1b68579b-9706-4efb-9d2e-d320d3aa4529:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:1b68579b-9706-4efb-9d2e-d320d3aa4529:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 361e2104 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 361e2104 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 361e2104 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4259 (Fri Sep 7 01:41:56 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4259 (Fri Sep 7 01:41:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "0637daa0", "local_conf_timestamp": 4259, "host-ts": 4259}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4249 (Fri Sep 7 01:41:46 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4249 (Fri Sep 7 01:41:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e3eb786c", "local_conf_timestamp": 4249, "host-ts": 4249}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:5ad0ec90-37c8-4156-9835-f6db4490d54f:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:5ad0ec90-37c8-4156-9835-f6db4490d54f:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 3967a538 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 3967a538 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 3967a538 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4259 (Fri Sep 7 01:41:56 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4259 (Fri Sep 7 01:41:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "0637daa0", "local_conf_timestamp": 4259, "host-ts": 4259}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4269 (Fri Sep 7 01:42:06 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4269 (Fri Sep 7 01:42:06 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "eb668666", "local_conf_timestamp": 4269, "host-ts": 4269}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:c677703e-de42-499e-946b-6bb80d527e9b:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:c677703e-de42-499e-946b-6bb80d527e9b:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 3d2401bc on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 3d2401bc on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 3d2401bc on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4269 (Fri Sep 7 01:42:07 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4269 (Fri Sep 7 01:42:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2f139fa7", "local_conf_timestamp": 4269, "host-ts": 4269}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4269 (Fri Sep 7 01:42:06 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4269 (Fri Sep 7 01:42:06 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "eb668666", "local_conf_timestamp": 4269, "host-ts": 4269}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:5fd905d8-2a2b-46d3-a93a-9d1ce753aa2e:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:5fd905d8-2a2b-46d3-a93a-9d1ce753aa2e:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 400293c6 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 400293c6 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 400293c6 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4269 (Fri Sep 7 01:42:07 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4269 (Fri Sep 7 01:42:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2f139fa7", "local_conf_timestamp": 4269, "host-ts": 4269}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4269 (Fri Sep 7 01:42:06 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4269 (Fri Sep 7 01:42:06 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "eb668666", "local_conf_timestamp": 4269, "host-ts": 4269}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:435cf0a2-abce-4456-8259-1c54be3322bf:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:435cf0a2-abce-4456-8259-1c54be3322bf:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 42e032ba on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 42e032ba on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 42e032ba on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4279 (Fri Sep 7 01:42:16 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=4279 (Fri Sep 7 01:42:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "aa89fffb", "local_conf_timestamp": 4279, "host-ts": 4279}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4279 (Fri Sep 7 01:42:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4279 (Fri Sep 7 01:42:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "725c7d7f", "local_conf_timestamp": 4279, "host-ts": 4279}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:d712d611-7633-442e-ba76-7b8a7b151650:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:d712d611-7633-442e-ba76-7b8a7b151650:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 43f4ef60 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 43f4ef60 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 43f4ef60 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4279 (Fri Sep 7 01:42:16 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=4279 (Fri Sep 7 01:42:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "aa89fffb", "local_conf_timestamp": 4279, "host-ts": 4279}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4279 (Fri Sep 7 01:42:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4279 (Fri Sep 7 01:42:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "725c7d7f", "local_conf_timestamp": 4279, "host-ts": 4279}, "global_maintenance": false}\n\nroot: INFO: * Engine has migrated.\nroot: INFO: * Waiting For System Stability...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/289/ Build Number: 289 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #285 [Gal Ben Haim] Adding dr suite [Barak Korren] Remove the populate_mock function from mock_runner [Daniel Belenky] mock_runner: store shell cmd in a variable [Barak Korren] mock_runner: Added timeout param [Barak Korren] Make whitelist repo configurable via env vars [Daniel Belenky] stdci_runner: let mock_runner manage timeout [Greg Sheremeta] remove 4.1 change queue from ovirt-engine-nodejs config Changes for Build #286 [Gal Ben Haim] Adding dr suite Changes for Build #287 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file [Barak Korren] Enable CI for kubevirt/client-python [Daniel Belenky] Add timeout config to stdci dsl [Daniel Belenky] stdci_runner.groovy: utilize DSL's timeout cfg Changes for Build #288 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #289 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 010_local_maintenance_cli.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... lago.ssh: DEBUG: start task:59794980-0644-429b-8e67-45b6d65ef738:Get ssh client for lago-he-node-ng-suite-4-2-host-1: lago.ssh: DEBUG: end task:59794980-0644-429b-8e67-45b6d65ef738:Get ssh client for lago-he-node-ng-suite-4-2-host-1: lago.ssh: DEBUG: Running 143e1d9e on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 143e1d9e on lago-he-node-ng-suite-4-2-host-1 returned with 0 lago.ssh: DEBUG: Command 143e1d9e on lago-he-node-ng-suite-4-2-host-1 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4097 (Fri Sep 7 09:40:23 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4098 (Fri Sep 7 09:40:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "30d99f17", "local_conf_timestamp": 4098, "host-ts": 4097}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4106 (Fri Sep 7 09:40:31 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4106 (Fri Sep 7 09:40:31 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f4595158", "local_conf_timestamp": 4106, "host-ts": 4106}, "global_maintenance": false} lago.ssh: DEBUG: start task:5f440943-57f2-4cce-a3ca-bc90febb599a:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:5f440943-57f2-4cce-a3ca-bc90febb599a:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 154731d0 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local lago.ssh: DEBUG: Command 154731d0 on lago-he-node-ng-suite-4-2-host-0 returned with 0 root: INFO: * Waiting for engine to migrate... lago.ssh: DEBUG: start task:f6a23462-6b67-4a7f-a8e5-ce3a1a61b9b9:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:f6a23462-6b67-4a7f-a8e5-ce3a1a61b9b9:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 16583696 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 16583696 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 16583696 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4117 (Fri Sep 7 09:40:43 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=4118 (Fri Sep 7 09:40:43 2018)\nconf_on_shared_storage=True\nmaintenance=True\nstate=LocalMaintenanceMigrateVm\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 0, "stopped": false, "maintenance": true, "crc32": "6686f364", "local_conf_timestamp": 4118, "host-ts": 4117}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4116 (Fri Sep 7 09:40:41 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4116 (Fri Sep 7 09:40:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f01fae5d", "local_conf_timestamp": 4116, "host-ts": 4116}, "global_maintenance": false} lago.ssh: DEBUG: start task:419d2e4d-095c-4a3b-88d5-8381583eface:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:419d2e4d-095c-4a3b-88d5-8381583eface:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 1dbb321c on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 1dbb321c on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 1dbb321c on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4128 (Fri Sep 7 09:40:54 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4128 (Fri Sep 7 09:40:54 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "86555762", "local_conf_timestamp": 4128, "host-ts": 4128}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4126 (Fri Sep 7 09:40:51 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4126 (Fri Sep 7 09:40:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fcd4af52", "local_conf_timestamp": 4126, "host-ts": 4126}, "global_maintenance": false} lago.ssh: DEBUG: start task:61fc83ab-70ae-413a-8df4-c9bb67758265:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:61fc83ab-70ae-413a-8df4-c9bb67758265:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 20b34fe0 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 20b34fe0 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 20b34fe0 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4128 (Fri Sep 7 09:40:54 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4128 (Fri Sep 7 09:40:54 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "86555762", "local_conf_timestamp": 4128, "host-ts": 4128}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4126 (Fri Sep 7 09:40:51 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4126 (Fri Sep 7 09:40:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fcd4af52", "local_conf_timestamp": 4126, "host-ts": 4126}, "global_maintenance": false} lago.ssh: DEBUG: start task:3d9aea5e-24c2-4068-9a10-ba518162f381:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:3d9aea5e-24c2-4068-9a10-ba518162f381:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 23c46cbe on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 23c46cbe on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 23c46cbe on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4138 (Fri Sep 7 09:41:04 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4139 (Fri Sep 7 09:41:04 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "5f615936", "local_conf_timestamp": 4139, "host-ts": 4138}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4136 (Fri Sep 7 09:41:02 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4136 (Fri Sep 7 09:41:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8faf65cf", "local_conf_timestamp": 4136, "host-ts": 4136}, "global_maintenance": false} lago.ssh: DEBUG: start task:ec4b43dc-c783-478e-8a31-112a3a29a98b:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:ec4b43dc-c783-478e-8a31-112a3a29a98b:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 26c15936 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 26c15936 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 26c15936 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4138 (Fri Sep 7 09:41:04 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4139 (Fri Sep 7 09:41:04 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "5f615936", "local_conf_timestamp": 4139, "host-ts": 4138}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4136 (Fri Sep 7 09:41:02 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4136 (Fri Sep 7 09:41:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8faf65cf", "local_conf_timestamp": 4136, "host-ts": 4136}, "global_maintenance": false} lago.ssh: DEBUG: start task:0213f742-5b86-4e60-a741-f7c2ad4909fc:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:0213f742-5b86-4e60-a741-f7c2ad4909fc:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 29d71b7e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 29d71b7e on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 29d71b7e on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4148 (Fri Sep 7 09:41:14 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4148 (Fri Sep 7 09:41:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d41ddd6c", "local_conf_timestamp": 4148, "host-ts": 4148}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4146 (Fri Sep 7 09:41:11 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4146 (Fri Sep 7 09:41:11 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "927f98d4", "local_conf_timestamp": 4146, "host-ts": 4146}, "global_maintenance": false} lago.ssh: DEBUG: start task:1bd02262-7217-4d9b-9283-a9e004820417:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:1bd02262-7217-4d9b-9283-a9e004820417:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 2cba09be on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 2cba09be on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 2cba09be on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4148 (Fri Sep 7 09:41:14 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4148 (Fri Sep 7 09:41:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d41ddd6c", "local_conf_timestamp": 4148, "host-ts": 4148}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4146 (Fri Sep 7 09:41:11 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4146 (Fri Sep 7 09:41:11 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "927f98d4", "local_conf_timestamp": 4146, "host-ts": 4146}, "global_maintenance": false} lago.ssh: DEBUG: start task:a0abfcf2-086c-4a46-9202-71064882839c:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:a0abfcf2-086c-4a46-9202-71064882839c:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 2fba5808 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 2fba5808 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 2fba5808 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4158 (Fri Sep 7 09:41:24 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4159 (Fri Sep 7 09:41:25 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "0d29d338", "local_conf_timestamp": 4159, "host-ts": 4158}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4156 (Fri Sep 7 09:41:21 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4156 (Fri Sep 7 09:41:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "251dbf5f", "local_conf_timestamp": 4156, "host-ts": 4156}, "global_maintenance": false} lago.ssh: DEBUG: start task:bb144426-dcfd-4b69-8e2a-fa361e2b5820:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:bb144426-dcfd-4b69-8e2a-fa361e2b5820:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 329e4be2 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 329e4be2 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 329e4be2 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4158 (Fri Sep 7 09:41:24 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4159 (Fri Sep 7 09:41:25 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "0d29d338", "local_conf_timestamp": 4159, "host-ts": 4158}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4156 (Fri Sep 7 09:41:21 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4156 (Fri Sep 7 09:41:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "251dbf5f", "local_conf_timestamp": 4156, "host-ts": 4156}, "global_maintenance": false} lago.ssh: DEBUG: start task:2a99382f-2658-4e7a-91e4-c76071ef5433:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:2a99382f-2658-4e7a-91e4-c76071ef5433:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 358c2176 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 358c2176 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 358c2176 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4169 (Fri Sep 7 09:41:35 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4169 (Fri Sep 7 09:41:35 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "b15e2e41", "local_conf_timestamp": 4169, "host-ts": 4169}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4166 (Fri Sep 7 09:41:31 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4166 (Fri Sep 7 09:41:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "29d6be50", "local_conf_timestamp": 4166, "host-ts": 4166}, "global_maintenance": false} lago.ssh: DEBUG: start task:94496173-9355-4141-b88f-cc115a433655:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:94496173-9355-4141-b88f-cc115a433655:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 38c5dc56 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 38c5dc56 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 38c5dc56 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4169 (Fri Sep 7 09:41:35 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4169 (Fri Sep 7 09:41:35 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "b15e2e41", "local_conf_timestamp": 4169, "host-ts": 4169}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4166 (Fri Sep 7 09:41:31 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4166 (Fri Sep 7 09:41:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "29d6be50", "local_conf_timestamp": 4166, "host-ts": 4166}, "global_maintenance": false} lago.ssh: DEBUG: start task:ebeda72c-9c7f-4209-ae39-c217a3c26b4f:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:ebeda72c-9c7f-4209-ae39-c217a3c26b4f:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 3bf49afc on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 3bf49afc on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 3bf49afc on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4179 (Fri Sep 7 09:41:45 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4180 (Fri Sep 7 09:41:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "VM migrated away successfully", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "adf4c27b", "local_conf_timestamp": 4180, "host-ts": 4179}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4176 (Fri Sep 7 09:41:42 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4176 (Fri Sep 7 09:41:42 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2d904155", "local_conf_timestamp": 4176, "host-ts": 4176}, "global_maintenance": false} lago.ssh: DEBUG: start task:d9537d84-54ff-41b2-95f6-3b6072a39a21:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:d9537d84-54ff-41b2-95f6-3b6072a39a21:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 3d3d9c7e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 3d3d9c7e on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 3d3d9c7e on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4179 (Fri Sep 7 09:41:45 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4180 (Fri Sep 7 09:41:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "VM migrated away successfully", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "adf4c27b", "local_conf_timestamp": 4180, "host-ts": 4179}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4176 (Fri Sep 7 09:41:42 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4176 (Fri Sep 7 09:41:42 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2d904155", "local_conf_timestamp": 4176, "host-ts": 4176}, "global_maintenance": false} lago.ssh: DEBUG: start task:8828d23b-f63d-43f7-9ae6-a50aea3127e4:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:8828d23b-f63d-43f7-9ae6-a50aea3127e4:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 44455e9e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 44455e9e on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 44455e9e on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4190 (Fri Sep 7 09:41:56 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=4190 (Fri Sep 7 09:41:56 2018)\nconf_on_shared_storage=True\nmaintenance=True\nstate=LocalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 0, "stopped": false, "maintenance": true, "crc32": "7d599ee4", "local_conf_timestamp": 4190, "host-ts": 4190}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4187 (Fri Sep 7 09:41:52 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4187 (Fri Sep 7 09:41:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c4bdcfab", "local_conf_timestamp": 4187, "host-ts": 4187}, "global_maintenance": false} root: INFO: * Engine has migrated. root: INFO: * Waiting For System Stability... cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 125, in local_maintenance _wait_for_engine_migration(host, he_index, "bad", "Migration Destination") File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 45, in _wait_for_engine_migration time.sleep(wait_value) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nlago.ssh: DEBUG: start task:59794980-0644-429b-8e67-45b6d65ef738:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: end task:59794980-0644-429b-8e67-45b6d65ef738:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: Running 143e1d9e on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 143e1d9e on lago-he-node-ng-suite-4-2-host-1 returned with 0\nlago.ssh: DEBUG: Command 143e1d9e on lago-he-node-ng-suite-4-2-host-1 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4097 (Fri Sep 7 09:40:23 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=4098 (Fri Sep 7 09:40:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "30d99f17", "local_conf_timestamp": 4098, "host-ts": 4097}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4106 (Fri Sep 7 09:40:31 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4106 (Fri Sep 7 09:40:31 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f4595158", "local_conf_timestamp": 4106, "host-ts": 4106}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:5f440943-57f2-4cce-a3ca-bc90febb599a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:5f440943-57f2-4cce-a3ca-bc90febb599a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 154731d0 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local\nlago.ssh: DEBUG: Command 154731d0 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nroot: INFO: * Waiting for engine to migrate...\nlago.ssh: DEBUG: start task:f6a23462-6b67-4a7f-a8e5-ce3a1a61b9b9:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:f6a23462-6b67-4a7f-a8e5-ce3a1a61b9b9:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 16583696 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 16583696 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 16583696 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4117 (Fri Sep 7 09:40:43 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=4118 (Fri Sep 7 09:40:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=True\\nstate=LocalMaintenanceMigrateVm\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 0, "stopped": false, "maintenance": true, "crc32": "6686f364", "local_conf_timestamp": 4118, "host-ts": 4117}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4116 (Fri Sep 7 09:40:41 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4116 (Fri Sep 7 09:40:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f01fae5d", "local_conf_timestamp": 4116, "host-ts": 4116}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:419d2e4d-095c-4a3b-88d5-8381583eface:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:419d2e4d-095c-4a3b-88d5-8381583eface:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 1dbb321c on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 1dbb321c on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 1dbb321c on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4128 (Fri Sep 7 09:40:54 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4128 (Fri Sep 7 09:40:54 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "86555762", "local_conf_timestamp": 4128, "host-ts": 4128}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4126 (Fri Sep 7 09:40:51 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4126 (Fri Sep 7 09:40:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fcd4af52", "local_conf_timestamp": 4126, "host-ts": 4126}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:61fc83ab-70ae-413a-8df4-c9bb67758265:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:61fc83ab-70ae-413a-8df4-c9bb67758265:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 20b34fe0 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 20b34fe0 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 20b34fe0 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4128 (Fri Sep 7 09:40:54 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4128 (Fri Sep 7 09:40:54 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "86555762", "local_conf_timestamp": 4128, "host-ts": 4128}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4126 (Fri Sep 7 09:40:51 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4126 (Fri Sep 7 09:40:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fcd4af52", "local_conf_timestamp": 4126, "host-ts": 4126}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:3d9aea5e-24c2-4068-9a10-ba518162f381:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:3d9aea5e-24c2-4068-9a10-ba518162f381:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 23c46cbe on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 23c46cbe on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 23c46cbe on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4138 (Fri Sep 7 09:41:04 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4139 (Fri Sep 7 09:41:04 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "5f615936", "local_conf_timestamp": 4139, "host-ts": 4138}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4136 (Fri Sep 7 09:41:02 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4136 (Fri Sep 7 09:41:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8faf65cf", "local_conf_timestamp": 4136, "host-ts": 4136}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:ec4b43dc-c783-478e-8a31-112a3a29a98b:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:ec4b43dc-c783-478e-8a31-112a3a29a98b:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 26c15936 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 26c15936 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 26c15936 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4138 (Fri Sep 7 09:41:04 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4139 (Fri Sep 7 09:41:04 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "5f615936", "local_conf_timestamp": 4139, "host-ts": 4138}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4136 (Fri Sep 7 09:41:02 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4136 (Fri Sep 7 09:41:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8faf65cf", "local_conf_timestamp": 4136, "host-ts": 4136}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:0213f742-5b86-4e60-a741-f7c2ad4909fc:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:0213f742-5b86-4e60-a741-f7c2ad4909fc:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 29d71b7e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 29d71b7e on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 29d71b7e on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4148 (Fri Sep 7 09:41:14 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4148 (Fri Sep 7 09:41:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d41ddd6c", "local_conf_timestamp": 4148, "host-ts": 4148}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4146 (Fri Sep 7 09:41:11 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4146 (Fri Sep 7 09:41:11 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "927f98d4", "local_conf_timestamp": 4146, "host-ts": 4146}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:1bd02262-7217-4d9b-9283-a9e004820417:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:1bd02262-7217-4d9b-9283-a9e004820417:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 2cba09be on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 2cba09be on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 2cba09be on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4148 (Fri Sep 7 09:41:14 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4148 (Fri Sep 7 09:41:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d41ddd6c", "local_conf_timestamp": 4148, "host-ts": 4148}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4146 (Fri Sep 7 09:41:11 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4146 (Fri Sep 7 09:41:11 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "927f98d4", "local_conf_timestamp": 4146, "host-ts": 4146}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:a0abfcf2-086c-4a46-9202-71064882839c:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:a0abfcf2-086c-4a46-9202-71064882839c:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 2fba5808 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 2fba5808 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 2fba5808 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4158 (Fri Sep 7 09:41:24 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4159 (Fri Sep 7 09:41:25 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "0d29d338", "local_conf_timestamp": 4159, "host-ts": 4158}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4156 (Fri Sep 7 09:41:21 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4156 (Fri Sep 7 09:41:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "251dbf5f", "local_conf_timestamp": 4156, "host-ts": 4156}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:bb144426-dcfd-4b69-8e2a-fa361e2b5820:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:bb144426-dcfd-4b69-8e2a-fa361e2b5820:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 329e4be2 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 329e4be2 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 329e4be2 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4158 (Fri Sep 7 09:41:24 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4159 (Fri Sep 7 09:41:25 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "0d29d338", "local_conf_timestamp": 4159, "host-ts": 4158}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4156 (Fri Sep 7 09:41:21 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4156 (Fri Sep 7 09:41:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "251dbf5f", "local_conf_timestamp": 4156, "host-ts": 4156}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:2a99382f-2658-4e7a-91e4-c76071ef5433:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:2a99382f-2658-4e7a-91e4-c76071ef5433:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 358c2176 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 358c2176 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 358c2176 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4169 (Fri Sep 7 09:41:35 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4169 (Fri Sep 7 09:41:35 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "b15e2e41", "local_conf_timestamp": 4169, "host-ts": 4169}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4166 (Fri Sep 7 09:41:31 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4166 (Fri Sep 7 09:41:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "29d6be50", "local_conf_timestamp": 4166, "host-ts": 4166}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:94496173-9355-4141-b88f-cc115a433655:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:94496173-9355-4141-b88f-cc115a433655:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 38c5dc56 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 38c5dc56 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 38c5dc56 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4169 (Fri Sep 7 09:41:35 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4169 (Fri Sep 7 09:41:35 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "b15e2e41", "local_conf_timestamp": 4169, "host-ts": 4169}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4166 (Fri Sep 7 09:41:31 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4166 (Fri Sep 7 09:41:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "29d6be50", "local_conf_timestamp": 4166, "host-ts": 4166}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:ebeda72c-9c7f-4209-ae39-c217a3c26b4f:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:ebeda72c-9c7f-4209-ae39-c217a3c26b4f:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 3bf49afc on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 3bf49afc on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 3bf49afc on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4179 (Fri Sep 7 09:41:45 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4180 (Fri Sep 7 09:41:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "VM migrated away successfully", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "adf4c27b", "local_conf_timestamp": 4180, "host-ts": 4179}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4176 (Fri Sep 7 09:41:42 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4176 (Fri Sep 7 09:41:42 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2d904155", "local_conf_timestamp": 4176, "host-ts": 4176}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:d9537d84-54ff-41b2-95f6-3b6072a39a21:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:d9537d84-54ff-41b2-95f6-3b6072a39a21:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 3d3d9c7e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 3d3d9c7e on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 3d3d9c7e on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4179 (Fri Sep 7 09:41:45 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4180 (Fri Sep 7 09:41:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "VM migrated away successfully", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "adf4c27b", "local_conf_timestamp": 4180, "host-ts": 4179}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4176 (Fri Sep 7 09:41:42 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4176 (Fri Sep 7 09:41:42 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2d904155", "local_conf_timestamp": 4176, "host-ts": 4176}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:8828d23b-f63d-43f7-9ae6-a50aea3127e4:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:8828d23b-f63d-43f7-9ae6-a50aea3127e4:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 44455e9e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 44455e9e on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 44455e9e on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4190 (Fri Sep 7 09:41:56 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=4190 (Fri Sep 7 09:41:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=True\\nstate=LocalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 0, "stopped": false, "maintenance": true, "crc32": "7d599ee4", "local_conf_timestamp": 4190, "host-ts": 4190}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4187 (Fri Sep 7 09:41:52 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4187 (Fri Sep 7 09:41:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c4bdcfab", "local_conf_timestamp": 4187, "host-ts": 4187}, "global_maintenance": false}\n\nroot: INFO: * Engine has migrated.\nroot: INFO: * Waiting For System Stability...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/290/ Build Number: 290 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #285 [Gal Ben Haim] Adding dr suite [Barak Korren] Remove the populate_mock function from mock_runner [Daniel Belenky] mock_runner: store shell cmd in a variable [Barak Korren] mock_runner: Added timeout param [Barak Korren] Make whitelist repo configurable via env vars [Daniel Belenky] stdci_runner: let mock_runner manage timeout [Greg Sheremeta] remove 4.1 change queue from ovirt-engine-nodejs config Changes for Build #286 [Gal Ben Haim] Adding dr suite Changes for Build #287 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file [Barak Korren] Enable CI for kubevirt/client-python [Daniel Belenky] Add timeout config to stdci dsl [Daniel Belenky] stdci_runner.groovy: utilize DSL's timeout cfg Changes for Build #288 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #289 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #290 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 012_local_maintenance_sdk.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/012_local_maintenance_sdk.py", line 42, in local_maintenance time.sleep(wait_value) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/291/ Build Number: 291 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #285 [Gal Ben Haim] Adding dr suite [Barak Korren] Remove the populate_mock function from mock_runner [Daniel Belenky] mock_runner: store shell cmd in a variable [Barak Korren] mock_runner: Added timeout param [Barak Korren] Make whitelist repo configurable via env vars [Daniel Belenky] stdci_runner: let mock_runner manage timeout [Greg Sheremeta] remove 4.1 change queue from ovirt-engine-nodejs config Changes for Build #286 [Gal Ben Haim] Adding dr suite Changes for Build #287 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file [Barak Korren] Enable CI for kubevirt/client-python [Daniel Belenky] Add timeout config to stdci dsl [Daniel Belenky] stdci_runner.groovy: utilize DSL's timeout cfg Changes for Build #288 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #289 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #290 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #291 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 012_local_maintenance_sdk.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/012_local_maintenance_sdk.py", line 42, in local_maintenance time.sleep(wait_value) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/292/ Build Number: 292 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #285 [Gal Ben Haim] Adding dr suite [Barak Korren] Remove the populate_mock function from mock_runner [Daniel Belenky] mock_runner: store shell cmd in a variable [Barak Korren] mock_runner: Added timeout param [Barak Korren] Make whitelist repo configurable via env vars [Daniel Belenky] stdci_runner: let mock_runner manage timeout [Greg Sheremeta] remove 4.1 change queue from ovirt-engine-nodejs config Changes for Build #286 [Gal Ben Haim] Adding dr suite Changes for Build #287 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file [Barak Korren] Enable CI for kubevirt/client-python [Daniel Belenky] Add timeout config to stdci dsl [Daniel Belenky] stdci_runner.groovy: utilize DSL's timeout cfg Changes for Build #288 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #289 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #290 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #291 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #292 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 012_local_maintenance_sdk.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/012_local_maintenance_sdk.py", line 42, in local_maintenance time.sleep(wait_value) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/293/ Build Number: 293 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #285 [Gal Ben Haim] Adding dr suite [Barak Korren] Remove the populate_mock function from mock_runner [Daniel Belenky] mock_runner: store shell cmd in a variable [Barak Korren] mock_runner: Added timeout param [Barak Korren] Make whitelist repo configurable via env vars [Daniel Belenky] stdci_runner: let mock_runner manage timeout [Greg Sheremeta] remove 4.1 change queue from ovirt-engine-nodejs config Changes for Build #286 [Gal Ben Haim] Adding dr suite Changes for Build #287 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file [Barak Korren] Enable CI for kubevirt/client-python [Daniel Belenky] Add timeout config to stdci dsl [Daniel Belenky] stdci_runner.groovy: utilize DSL's timeout cfg Changes for Build #288 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #289 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #290 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #291 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #292 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #293 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 012_local_maintenance_sdk.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... root: INFO: * Performing Deactivation... cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/012_local_maintenance_sdk.py", line 62, in local_maintenance host_service.deactivate() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 36838, in deactivate return self._internal_action(action, 'deactivate', None, headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 299, in _internal_action return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 54, in wait response = self._connection.wait(self._context) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 496, in wait return self.__wait(context, failed_auth) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 532, in __wait self._multi.select(1.0) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nroot: INFO: * Performing Deactivation...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/294/ Build Number: 294 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #285 [Gal Ben Haim] Adding dr suite [Barak Korren] Remove the populate_mock function from mock_runner [Daniel Belenky] mock_runner: store shell cmd in a variable [Barak Korren] mock_runner: Added timeout param [Barak Korren] Make whitelist repo configurable via env vars [Daniel Belenky] stdci_runner: let mock_runner manage timeout [Greg Sheremeta] remove 4.1 change queue from ovirt-engine-nodejs config Changes for Build #286 [Gal Ben Haim] Adding dr suite Changes for Build #287 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file [Barak Korren] Enable CI for kubevirt/client-python [Daniel Belenky] Add timeout config to stdci dsl [Daniel Belenky] stdci_runner.groovy: utilize DSL's timeout cfg Changes for Build #288 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #289 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #290 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #291 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #292 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #293 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #294 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 012_local_maintenance_sdk.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/012_local_maintenance_sdk.py", line 42, in local_maintenance time.sleep(wait_value) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/295/ Build Number: 295 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #285 [Gal Ben Haim] Adding dr suite [Barak Korren] Remove the populate_mock function from mock_runner [Daniel Belenky] mock_runner: store shell cmd in a variable [Barak Korren] mock_runner: Added timeout param [Barak Korren] Make whitelist repo configurable via env vars [Daniel Belenky] stdci_runner: let mock_runner manage timeout [Greg Sheremeta] remove 4.1 change queue from ovirt-engine-nodejs config Changes for Build #286 [Gal Ben Haim] Adding dr suite Changes for Build #287 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file [Barak Korren] Enable CI for kubevirt/client-python [Daniel Belenky] Add timeout config to stdci dsl [Daniel Belenky] stdci_runner.groovy: utilize DSL's timeout cfg Changes for Build #288 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #289 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #290 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #291 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #292 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #293 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #294 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #295 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 012_local_maintenance_sdk.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... root: INFO: * Performing Deactivation... root: INFO: * Performing Activation... root: INFO: * Waiting For System Stability... cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/012_local_maintenance_sdk.py", line 79, in local_maintenance time.sleep(wait_value) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nroot: INFO: * Performing Deactivation...\nroot: INFO: * Performing Activation...\nroot: INFO: * Waiting For System Stability...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/296/ Build Number: 296 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #285 [Gal Ben Haim] Adding dr suite [Barak Korren] Remove the populate_mock function from mock_runner [Daniel Belenky] mock_runner: store shell cmd in a variable [Barak Korren] mock_runner: Added timeout param [Barak Korren] Make whitelist repo configurable via env vars [Daniel Belenky] stdci_runner: let mock_runner manage timeout [Greg Sheremeta] remove 4.1 change queue from ovirt-engine-nodejs config Changes for Build #286 [Gal Ben Haim] Adding dr suite Changes for Build #287 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file [Barak Korren] Enable CI for kubevirt/client-python [Daniel Belenky] Add timeout config to stdci dsl [Daniel Belenky] stdci_runner.groovy: utilize DSL's timeout cfg Changes for Build #288 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #289 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #290 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #291 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #292 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #293 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #294 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #295 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #296 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 010_local_maintenance_cli.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... lago.ssh: DEBUG: start task:edabe519-380e-4f24-8ccc-6039cb0e9862:Get ssh client for lago-he-node-ng-suite-4-2-host-1: lago.ssh: DEBUG: end task:edabe519-380e-4f24-8ccc-6039cb0e9862:Get ssh client for lago-he-node-ng-suite-4-2-host-1: lago.ssh: DEBUG: Running 924fd188 on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 924fd188 on lago-he-node-ng-suite-4-2-host-1 returned with 0 lago.ssh: DEBUG: Command 924fd188 on lago-he-node-ng-suite-4-2-host-1 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4123 (Sun Sep 9 17:41:17 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4123 (Sun Sep 9 17:41:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "542fda3d", "local_conf_timestamp": 4123, "host-ts": 4123}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4123 (Sun Sep 9 17:41:17 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4123 (Sun Sep 9 17:41:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7bfaf70f", "local_conf_timestamp": 4123, "host-ts": 4123}, "global_maintenance": false} lago.ssh: DEBUG: start task:70d9267e-886f-466a-8307-0f400e1aa238:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:70d9267e-886f-466a-8307-0f400e1aa238:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 936816ac on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local lago.ssh: DEBUG: Command 936816ac on lago-he-node-ng-suite-4-2-host-0 returned with 0 root: INFO: * Waiting for engine to migrate... lago.ssh: DEBUG: start task:e82b8660-b1bb-4876-af36-8cc0508f6e50:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:e82b8660-b1bb-4876-af36-8cc0508f6e50:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 948465e0 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 948465e0 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 948465e0 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4133 (Sun Sep 9 17:41:27 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4133 (Sun Sep 9 17:41:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ab27cd2e", "local_conf_timestamp": 4133, "host-ts": 4133}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4133 (Sun Sep 9 17:41:27 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4133 (Sun Sep 9 17:41:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7fbc080a", "local_conf_timestamp": 4133, "host-ts": 4133}, "global_maintenance": false} lago.ssh: DEBUG: start task:faae2a37-fa93-434e-b64f-c212493d509a:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:faae2a37-fa93-434e-b64f-c212493d509a:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 9cbb6ff6 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9cbb6ff6 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 9cbb6ff6 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4143 (Sun Sep 9 17:41:37 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4143 (Sun Sep 9 17:41:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "56f11c51", "local_conf_timestamp": 4143, "host-ts": 4143}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4143 (Sun Sep 9 17:41:37 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4143 (Sun Sep 9 17:41:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1551c089", "local_conf_timestamp": 4143, "host-ts": 4143}, "global_maintenance": false} lago.ssh: DEBUG: start task:3eb604ad-0c0f-49f5-bacf-40c93d189369:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:3eb604ad-0c0f-49f5-bacf-40c93d189369:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 9fd02326 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9fd02326 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 9fd02326 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4153 (Sun Sep 9 17:41:47 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4153 (Sun Sep 9 17:41:48 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f8c22293", "local_conf_timestamp": 4153, "host-ts": 4153}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4153 (Sun Sep 9 17:41:47 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4153 (Sun Sep 9 17:41:47 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "11173f8c", "local_conf_timestamp": 4153, "host-ts": 4153}, "global_maintenance": false} lago.ssh: DEBUG: start task:70f047d8-5dbe-4ff9-a686-15209e393c34:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:70f047d8-5dbe-4ff9-a686-15209e393c34:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running a2ca1b18 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a2ca1b18 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command a2ca1b18 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4153 (Sun Sep 9 17:41:47 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4153 (Sun Sep 9 17:41:48 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f8c22293", "local_conf_timestamp": 4153, "host-ts": 4153}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4153 (Sun Sep 9 17:41:47 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4153 (Sun Sep 9 17:41:47 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "11173f8c", "local_conf_timestamp": 4153, "host-ts": 4153}, "global_maintenance": false} lago.ssh: DEBUG: start task:497f2589-5665-48b6-b564-7c75ad74fc7e:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:497f2589-5665-48b6-b564-7c75ad74fc7e:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running a5ba5dce on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a5ba5dce on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command a5ba5dce on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4153 (Sun Sep 9 17:41:47 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4153 (Sun Sep 9 17:41:48 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f8c22293", "local_conf_timestamp": 4153, "host-ts": 4153}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4153 (Sun Sep 9 17:41:47 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4153 (Sun Sep 9 17:41:47 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "11173f8c", "local_conf_timestamp": 4153, "host-ts": 4153}, "global_maintenance": false} lago.ssh: DEBUG: start task:61ee37b7-c844-44bf-82f1-ec77a530e063:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:61ee37b7-c844-44bf-82f1-ec77a530e063:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running a8dae79e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a8dae79e on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command a8dae79e on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4163 (Sun Sep 9 17:41:58 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4163 (Sun Sep 9 17:41:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d1e66794", "local_conf_timestamp": 4163, "host-ts": 4163}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4163 (Sun Sep 9 17:41:57 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4163 (Sun Sep 9 17:41:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ddc3e83", "local_conf_timestamp": 4163, "host-ts": 4163}, "global_maintenance": false} lago.ssh: DEBUG: start task:e880b7f5-62fb-48ef-9f5e-aabaab7394fa:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:e880b7f5-62fb-48ef-9f5e-aabaab7394fa:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running ac1eeed2 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ac1eeed2 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command ac1eeed2 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4173 (Sun Sep 9 17:42:07 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4173 (Sun Sep 9 17:42:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7fd55956", "local_conf_timestamp": 4173, "host-ts": 4173}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4163 (Sun Sep 9 17:41:57 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4163 (Sun Sep 9 17:41:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ddc3e83", "local_conf_timestamp": 4163, "host-ts": 4163}, "global_maintenance": false} lago.ssh: DEBUG: start task:1c5a6117-78e6-4c6e-84d7-6d49707b45c0:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:1c5a6117-78e6-4c6e-84d7-6d49707b45c0:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running af1482fa on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command af1482fa on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command af1482fa on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4173 (Sun Sep 9 17:42:07 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4173 (Sun Sep 9 17:42:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7fd55956", "local_conf_timestamp": 4173, "host-ts": 4173}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4163 (Sun Sep 9 17:41:57 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4163 (Sun Sep 9 17:41:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ddc3e83", "local_conf_timestamp": 4163, "host-ts": 4163}, "global_maintenance": false} lago.ssh: DEBUG: start task:f3f27f9a-c470-4604-b660-77705ed49223:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:f3f27f9a-c470-4604-b660-77705ed49223:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running b27ad9f8 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b27ad9f8 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command b27ad9f8 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4183 (Sun Sep 9 17:42:17 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4183 (Sun Sep 9 17:42:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f260084d", "local_conf_timestamp": 4183, "host-ts": 4183}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4173 (Sun Sep 9 17:42:08 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4174 (Sun Sep 9 17:42:08 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "34da8cab", "local_conf_timestamp": 4174, "host-ts": 4173}, "global_maintenance": false} lago.ssh: DEBUG: start task:7acfa077-36eb-4952-8de2-3c48c7f2be7a:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:7acfa077-36eb-4952-8de2-3c48c7f2be7a:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running b588ed38 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b588ed38 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command b588ed38 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4183 (Sun Sep 9 17:42:17 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4183 (Sun Sep 9 17:42:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f260084d", "local_conf_timestamp": 4183, "host-ts": 4183}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4173 (Sun Sep 9 17:42:08 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4174 (Sun Sep 9 17:42:08 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "34da8cab", "local_conf_timestamp": 4174, "host-ts": 4173}, "global_maintenance": false} lago.ssh: DEBUG: start task:b7b4d91f-a18e-40b0-8f0c-ea30d7c27ebd:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:b7b4d91f-a18e-40b0-8f0c-ea30d7c27ebd:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running b8ce1824 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b8ce1824 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command b8ce1824 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4193 (Sun Sep 9 17:42:27 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4193 (Sun Sep 9 17:42:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "5c53368f", "local_conf_timestamp": 4193, "host-ts": 4193}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4184 (Sun Sep 9 17:42:18 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4184 (Sun Sep 9 17:42:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f2b4632b", "local_conf_timestamp": 4184, "host-ts": 4184}, "global_maintenance": false} lago.ssh: DEBUG: start task:770f319e-a58d-4eda-b78f-3140579dabc1:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:770f319e-a58d-4eda-b78f-3140579dabc1:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running bc35167a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command bc35167a on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command bc35167a on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4193 (Sun Sep 9 17:42:27 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4193 (Sun Sep 9 17:42:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "5c53368f", "local_conf_timestamp": 4193, "host-ts": 4193}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4184 (Sun Sep 9 17:42:18 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4184 (Sun Sep 9 17:42:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f2b4632b", "local_conf_timestamp": 4184, "host-ts": 4184}, "global_maintenance": false} lago.ssh: DEBUG: start task:27de6cdb-de0c-4276-9814-12506fa85728:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:27de6cdb-de0c-4276-9814-12506fa85728:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running bf21e69c on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command bf21e69c on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command bf21e69c on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4203 (Sun Sep 9 17:42:37 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4203 (Sun Sep 9 17:42:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "VM migrated away successfully", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2f58a03f", "local_conf_timestamp": 4203, "host-ts": 4203}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4194 (Sun Sep 9 17:42:28 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4194 (Sun Sep 9 17:42:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f6f29c2e", "local_conf_timestamp": 4194, "host-ts": 4194}, "global_maintenance": false} lago.ssh: DEBUG: start task:9f1737f3-0c07-4875-a7fa-b488f478fff5:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:9f1737f3-0c07-4875-a7fa-b488f478fff5:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running c03ece78 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command c03ece78 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command c03ece78 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4203 (Sun Sep 9 17:42:37 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4203 (Sun Sep 9 17:42:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "VM migrated away successfully", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2f58a03f", "local_conf_timestamp": 4203, "host-ts": 4203}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4194 (Sun Sep 9 17:42:28 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4194 (Sun Sep 9 17:42:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f6f29c2e", "local_conf_timestamp": 4194, "host-ts": 4194}, "global_maintenance": false} lago.ssh: DEBUG: start task:25dc3ef2-7e44-4aac-a95c-537ab99bfbfe:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:25dc3ef2-7e44-4aac-a95c-537ab99bfbfe:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running c75a9994 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command c75a9994 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command c75a9994 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4213 (Sun Sep 9 17:42:48 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=4213 (Sun Sep 9 17:42:48 2018)\nconf_on_shared_storage=True\nmaintenance=True\nstate=LocalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 0, "stopped": false, "maintenance": true, "crc32": "bf9ec4ae", "local_conf_timestamp": 4213, "host-ts": 4213}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4205 (Sun Sep 9 17:42:39 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4204 (Sun Sep 9 17:42:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5f60df5b", "local_conf_timestamp": 4204, "host-ts": 4205}, "global_maintenance": false} root: INFO: * Engine has migrated. root: INFO: * Waiting For System Stability... cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 125, in local_maintenance _wait_for_engine_migration(host, he_index, "bad", "Migration Destination") File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 45, in _wait_for_engine_migration time.sleep(wait_value) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nlago.ssh: DEBUG: start task:edabe519-380e-4f24-8ccc-6039cb0e9862:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: end task:edabe519-380e-4f24-8ccc-6039cb0e9862:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: Running 924fd188 on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 924fd188 on lago-he-node-ng-suite-4-2-host-1 returned with 0\nlago.ssh: DEBUG: Command 924fd188 on lago-he-node-ng-suite-4-2-host-1 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4123 (Sun Sep 9 17:41:17 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=4123 (Sun Sep 9 17:41:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "542fda3d", "local_conf_timestamp": 4123, "host-ts": 4123}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4123 (Sun Sep 9 17:41:17 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4123 (Sun Sep 9 17:41:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7bfaf70f", "local_conf_timestamp": 4123, "host-ts": 4123}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:70d9267e-886f-466a-8307-0f400e1aa238:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:70d9267e-886f-466a-8307-0f400e1aa238:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 936816ac on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local\nlago.ssh: DEBUG: Command 936816ac on lago-he-node-ng-suite-4-2-host-0 returned with 0\nroot: INFO: * Waiting for engine to migrate...\nlago.ssh: DEBUG: start task:e82b8660-b1bb-4876-af36-8cc0508f6e50:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:e82b8660-b1bb-4876-af36-8cc0508f6e50:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 948465e0 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 948465e0 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 948465e0 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4133 (Sun Sep 9 17:41:27 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4133 (Sun Sep 9 17:41:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ab27cd2e", "local_conf_timestamp": 4133, "host-ts": 4133}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4133 (Sun Sep 9 17:41:27 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4133 (Sun Sep 9 17:41:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7fbc080a", "local_conf_timestamp": 4133, "host-ts": 4133}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:faae2a37-fa93-434e-b64f-c212493d509a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:faae2a37-fa93-434e-b64f-c212493d509a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 9cbb6ff6 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9cbb6ff6 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 9cbb6ff6 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4143 (Sun Sep 9 17:41:37 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4143 (Sun Sep 9 17:41:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "56f11c51", "local_conf_timestamp": 4143, "host-ts": 4143}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4143 (Sun Sep 9 17:41:37 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4143 (Sun Sep 9 17:41:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1551c089", "local_conf_timestamp": 4143, "host-ts": 4143}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:3eb604ad-0c0f-49f5-bacf-40c93d189369:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:3eb604ad-0c0f-49f5-bacf-40c93d189369:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 9fd02326 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9fd02326 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 9fd02326 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4153 (Sun Sep 9 17:41:47 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4153 (Sun Sep 9 17:41:48 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f8c22293", "local_conf_timestamp": 4153, "host-ts": 4153}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4153 (Sun Sep 9 17:41:47 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4153 (Sun Sep 9 17:41:47 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "11173f8c", "local_conf_timestamp": 4153, "host-ts": 4153}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:70f047d8-5dbe-4ff9-a686-15209e393c34:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:70f047d8-5dbe-4ff9-a686-15209e393c34:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running a2ca1b18 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a2ca1b18 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command a2ca1b18 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4153 (Sun Sep 9 17:41:47 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4153 (Sun Sep 9 17:41:48 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f8c22293", "local_conf_timestamp": 4153, "host-ts": 4153}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4153 (Sun Sep 9 17:41:47 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4153 (Sun Sep 9 17:41:47 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "11173f8c", "local_conf_timestamp": 4153, "host-ts": 4153}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:497f2589-5665-48b6-b564-7c75ad74fc7e:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:497f2589-5665-48b6-b564-7c75ad74fc7e:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running a5ba5dce on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a5ba5dce on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command a5ba5dce on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4153 (Sun Sep 9 17:41:47 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4153 (Sun Sep 9 17:41:48 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f8c22293", "local_conf_timestamp": 4153, "host-ts": 4153}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4153 (Sun Sep 9 17:41:47 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4153 (Sun Sep 9 17:41:47 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "11173f8c", "local_conf_timestamp": 4153, "host-ts": 4153}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:61ee37b7-c844-44bf-82f1-ec77a530e063:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:61ee37b7-c844-44bf-82f1-ec77a530e063:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running a8dae79e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a8dae79e on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command a8dae79e on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4163 (Sun Sep 9 17:41:58 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4163 (Sun Sep 9 17:41:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d1e66794", "local_conf_timestamp": 4163, "host-ts": 4163}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4163 (Sun Sep 9 17:41:57 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4163 (Sun Sep 9 17:41:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ddc3e83", "local_conf_timestamp": 4163, "host-ts": 4163}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:e880b7f5-62fb-48ef-9f5e-aabaab7394fa:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:e880b7f5-62fb-48ef-9f5e-aabaab7394fa:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running ac1eeed2 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ac1eeed2 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command ac1eeed2 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4173 (Sun Sep 9 17:42:07 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4173 (Sun Sep 9 17:42:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7fd55956", "local_conf_timestamp": 4173, "host-ts": 4173}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4163 (Sun Sep 9 17:41:57 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4163 (Sun Sep 9 17:41:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ddc3e83", "local_conf_timestamp": 4163, "host-ts": 4163}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:1c5a6117-78e6-4c6e-84d7-6d49707b45c0:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:1c5a6117-78e6-4c6e-84d7-6d49707b45c0:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running af1482fa on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command af1482fa on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command af1482fa on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4173 (Sun Sep 9 17:42:07 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4173 (Sun Sep 9 17:42:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7fd55956", "local_conf_timestamp": 4173, "host-ts": 4173}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4163 (Sun Sep 9 17:41:57 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4163 (Sun Sep 9 17:41:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ddc3e83", "local_conf_timestamp": 4163, "host-ts": 4163}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:f3f27f9a-c470-4604-b660-77705ed49223:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:f3f27f9a-c470-4604-b660-77705ed49223:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b27ad9f8 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b27ad9f8 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command b27ad9f8 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4183 (Sun Sep 9 17:42:17 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4183 (Sun Sep 9 17:42:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f260084d", "local_conf_timestamp": 4183, "host-ts": 4183}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4173 (Sun Sep 9 17:42:08 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4174 (Sun Sep 9 17:42:08 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "34da8cab", "local_conf_timestamp": 4174, "host-ts": 4173}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:7acfa077-36eb-4952-8de2-3c48c7f2be7a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:7acfa077-36eb-4952-8de2-3c48c7f2be7a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b588ed38 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b588ed38 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command b588ed38 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4183 (Sun Sep 9 17:42:17 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4183 (Sun Sep 9 17:42:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f260084d", "local_conf_timestamp": 4183, "host-ts": 4183}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4173 (Sun Sep 9 17:42:08 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4174 (Sun Sep 9 17:42:08 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "34da8cab", "local_conf_timestamp": 4174, "host-ts": 4173}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:b7b4d91f-a18e-40b0-8f0c-ea30d7c27ebd:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:b7b4d91f-a18e-40b0-8f0c-ea30d7c27ebd:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b8ce1824 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b8ce1824 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command b8ce1824 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4193 (Sun Sep 9 17:42:27 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4193 (Sun Sep 9 17:42:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "5c53368f", "local_conf_timestamp": 4193, "host-ts": 4193}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4184 (Sun Sep 9 17:42:18 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4184 (Sun Sep 9 17:42:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f2b4632b", "local_conf_timestamp": 4184, "host-ts": 4184}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:770f319e-a58d-4eda-b78f-3140579dabc1:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:770f319e-a58d-4eda-b78f-3140579dabc1:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running bc35167a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command bc35167a on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command bc35167a on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4193 (Sun Sep 9 17:42:27 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4193 (Sun Sep 9 17:42:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "5c53368f", "local_conf_timestamp": 4193, "host-ts": 4193}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4184 (Sun Sep 9 17:42:18 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4184 (Sun Sep 9 17:42:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f2b4632b", "local_conf_timestamp": 4184, "host-ts": 4184}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:27de6cdb-de0c-4276-9814-12506fa85728:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:27de6cdb-de0c-4276-9814-12506fa85728:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running bf21e69c on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command bf21e69c on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command bf21e69c on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4203 (Sun Sep 9 17:42:37 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=4203 (Sun Sep 9 17:42:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "VM migrated away successfully", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2f58a03f", "local_conf_timestamp": 4203, "host-ts": 4203}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4194 (Sun Sep 9 17:42:28 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4194 (Sun Sep 9 17:42:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f6f29c2e", "local_conf_timestamp": 4194, "host-ts": 4194}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:9f1737f3-0c07-4875-a7fa-b488f478fff5:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:9f1737f3-0c07-4875-a7fa-b488f478fff5:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running c03ece78 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command c03ece78 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command c03ece78 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4203 (Sun Sep 9 17:42:37 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=4203 (Sun Sep 9 17:42:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "VM migrated away successfully", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2f58a03f", "local_conf_timestamp": 4203, "host-ts": 4203}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4194 (Sun Sep 9 17:42:28 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4194 (Sun Sep 9 17:42:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f6f29c2e", "local_conf_timestamp": 4194, "host-ts": 4194}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:25dc3ef2-7e44-4aac-a95c-537ab99bfbfe:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:25dc3ef2-7e44-4aac-a95c-537ab99bfbfe:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running c75a9994 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command c75a9994 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command c75a9994 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4213 (Sun Sep 9 17:42:48 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=4213 (Sun Sep 9 17:42:48 2018)\\nconf_on_shared_storage=True\\nmaintenance=True\\nstate=LocalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 0, "stopped": false, "maintenance": true, "crc32": "bf9ec4ae", "local_conf_timestamp": 4213, "host-ts": 4213}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4205 (Sun Sep 9 17:42:39 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4204 (Sun Sep 9 17:42:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5f60df5b", "local_conf_timestamp": 4204, "host-ts": 4205}, "global_maintenance": false}\n\nroot: INFO: * Engine has migrated.\nroot: INFO: * Waiting For System Stability...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/297/ Build Number: 297 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #285 [Gal Ben Haim] Adding dr suite [Barak Korren] Remove the populate_mock function from mock_runner [Daniel Belenky] mock_runner: store shell cmd in a variable [Barak Korren] mock_runner: Added timeout param [Barak Korren] Make whitelist repo configurable via env vars [Daniel Belenky] stdci_runner: let mock_runner manage timeout [Greg Sheremeta] remove 4.1 change queue from ovirt-engine-nodejs config Changes for Build #286 [Gal Ben Haim] Adding dr suite Changes for Build #287 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file [Barak Korren] Enable CI for kubevirt/client-python [Daniel Belenky] Add timeout config to stdci dsl [Daniel Belenky] stdci_runner.groovy: utilize DSL's timeout cfg Changes for Build #288 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #289 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #290 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #291 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #292 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #293 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #294 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #295 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #296 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #297 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 012_local_maintenance_sdk.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/012_local_maintenance_sdk.py", line 42, in local_maintenance time.sleep(wait_value) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/298/ Build Number: 298 Build Status: Fixed Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #285 [Gal Ben Haim] Adding dr suite [Barak Korren] Remove the populate_mock function from mock_runner [Daniel Belenky] mock_runner: store shell cmd in a variable [Barak Korren] mock_runner: Added timeout param [Barak Korren] Make whitelist repo configurable via env vars [Daniel Belenky] stdci_runner: let mock_runner manage timeout [Greg Sheremeta] remove 4.1 change queue from ovirt-engine-nodejs config Changes for Build #286 [Gal Ben Haim] Adding dr suite Changes for Build #287 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file [Barak Korren] Enable CI for kubevirt/client-python [Daniel Belenky] Add timeout config to stdci dsl [Daniel Belenky] stdci_runner.groovy: utilize DSL's timeout cfg Changes for Build #288 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #289 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #290 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #291 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #292 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #293 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #294 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #295 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #296 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #297 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #298 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file [Yuval Turgeman] ovirt-node-ng: remove nightly master build ----------------- Failed Tests: ----------------- All tests passed
participants (1)
-
jenkins＠jenkins.phx.ovirt.org