Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/287/ Build Number: 287 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #285 [Gal Ben Haim] Adding dr suite [Barak Korren] Remove the populate_mock function from mock_runner [Daniel Belenky] mock_runner: store shell cmd in a variable [Barak Korren] mock_runner: Added timeout param [Barak Korren] Make whitelist repo configurable via env vars [Daniel Belenky] stdci_runner: let mock_runner manage timeout [Greg Sheremeta] remove 4.1 change queue from ovirt-engine-nodejs config Changes for Build #286 [Gal Ben Haim] Adding dr suite Changes for Build #287 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file [Barak Korren] Enable CI for kubevirt/client-python [Daniel Belenky] Add timeout config to stdci dsl [Daniel Belenky] stdci_runner.groovy: utilize DSL's timeout cfg ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 010_local_maintenance_cli.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... lago.ssh: DEBUG: start task:dbac6aee-f807-447a-8c86-15134f2b7749:Get ssh client for lago-he-node-ng-suite-4-2-host-1: lago.ssh: DEBUG: end task:dbac6aee-f807-447a-8c86-15134f2b7749:Get ssh client for lago-he-node-ng-suite-4-2-host-1: lago.ssh: DEBUG: Running daa5b834 on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json lago.ssh: DEBUG: Command daa5b834 on lago-he-node-ng-suite-4-2-host-1 returned with 0 lago.ssh: DEBUG: Command daa5b834 on lago-he-node-ng-suite-4-2-host-1 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4146 (Thu Sep 6 17:39:36 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4146 (Thu Sep 6 17:39:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e9066a29", "local_conf_timestamp": 4146, "host-ts": 4146}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4153 (Thu Sep 6 17:39:43 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4153 (Thu Sep 6 17:39:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "662a0a14", "local_conf_timestamp": 4153, "host-ts": 4153}, "global_maintenance": false} lago.ssh: DEBUG: start task:d30f82be-086c-4823-8307-6871f22336f6:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:d30f82be-086c-4823-8307-6871f22336f6:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running dbb9947a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local lago.ssh: DEBUG: Command dbb9947a on lago-he-node-ng-suite-4-2-host-0 returned with 0 root: INFO: * Waiting for engine to migrate... lago.ssh: DEBUG: start task:7c4ae3ab-7501-4c92-9bea-fdaaef4859b2:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:7c4ae3ab-7501-4c92-9bea-fdaaef4859b2:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running dce9aff6 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command dce9aff6 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command dce9aff6 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4156 (Thu Sep 6 17:39:46 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4156 (Thu Sep 6 17:39:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "35612a16", "local_conf_timestamp": 4156, "host-ts": 4156}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4153 (Thu Sep 6 17:39:43 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4153 (Thu Sep 6 17:39:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "662a0a14", "local_conf_timestamp": 4153, "host-ts": 4153}, "global_maintenance": false} lago.ssh: DEBUG: start task:eb74d1b1-2d4e-416b-a20f-1bc0ce171837:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:eb74d1b1-2d4e-416b-a20f-1bc0ce171837:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running e4cbecf2 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e4cbecf2 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command e4cbecf2 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4177 (Thu Sep 6 17:40:07 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4177 (Thu Sep 6 17:40:08 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "40657674", "local_conf_timestamp": 4177, "host-ts": 4177}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4173 (Thu Sep 6 17:40:03 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4174 (Thu Sep 6 17:40:04 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f0c361bd", "local_conf_timestamp": 4174, "host-ts": 4173}, "global_maintenance": false} lago.ssh: DEBUG: start task:9b2117bb-3e9b-4784-bb67-0d5f98f9d7b5:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:9b2117bb-3e9b-4784-bb67-0d5f98f9d7b5:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running e8606ab4 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e8606ab4 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command e8606ab4 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4177 (Thu Sep 6 17:40:07 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4177 (Thu Sep 6 17:40:08 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "40657674", "local_conf_timestamp": 4177, "host-ts": 4177}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4173 (Thu Sep 6 17:40:03 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4174 (Thu Sep 6 17:40:04 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f0c361bd", "local_conf_timestamp": 4174, "host-ts": 4173}, "global_maintenance": false} lago.ssh: DEBUG: start task:ddbde6de-1c5d-42c4-9156-a1c592cb6dd0:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:ddbde6de-1c5d-42c4-9156-a1c592cb6dd0:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running eb997522 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command eb997522 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command eb997522 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4187 (Thu Sep 6 17:40:17 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4187 (Thu Sep 6 17:40:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "cdd0276f", "local_conf_timestamp": 4187, "host-ts": 4187}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4183 (Thu Sep 6 17:40:13 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4183 (Thu Sep 6 17:40:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "267dc4b5", "local_conf_timestamp": 4183, "host-ts": 4183}, "global_maintenance": false} lago.ssh: DEBUG: start task:fc0e0ad5-af11-42b1-8b1a-542db357c10f:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:fc0e0ad5-af11-42b1-8b1a-542db357c10f:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running ee95607e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ee95607e on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command ee95607e on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4187 (Thu Sep 6 17:40:17 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4187 (Thu Sep 6 17:40:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "cdd0276f", "local_conf_timestamp": 4187, "host-ts": 4187}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4183 (Thu Sep 6 17:40:13 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4183 (Thu Sep 6 17:40:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "267dc4b5", "local_conf_timestamp": 4183, "host-ts": 4183}, "global_maintenance": false} lago.ssh: DEBUG: start task:33c75a20-69a6-4f1f-9d10-5964b70e5476:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:33c75a20-69a6-4f1f-9d10-5964b70e5476:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running f22070d0 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f22070d0 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command f22070d0 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4197 (Thu Sep 6 17:40:27 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4197 (Thu Sep 6 17:40:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "63e319ad", "local_conf_timestamp": 4197, "host-ts": 4197}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4193 (Thu Sep 6 17:40:23 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4193 (Thu Sep 6 17:40:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "223b3bb0", "local_conf_timestamp": 4193, "host-ts": 4193}, "global_maintenance": false} lago.ssh: DEBUG: start task:eebbdb9b-3cca-497b-83fb-90f5b365bd00:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:eebbdb9b-3cca-497b-83fb-90f5b365bd00:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running f5324a00 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f5324a00 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command f5324a00 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4197 (Thu Sep 6 17:40:27 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4197 (Thu Sep 6 17:40:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "63e319ad", "local_conf_timestamp": 4197, "host-ts": 4197}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4193 (Thu Sep 6 17:40:23 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4193 (Thu Sep 6 17:40:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "223b3bb0", "local_conf_timestamp": 4193, "host-ts": 4193}, "global_maintenance": false} lago.ssh: DEBUG: start task:901e9f22-e329-4658-a951-0e1aa80b3cd6:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:901e9f22-e329-4658-a951-0e1aa80b3cd6:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running f81411ae on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f81411ae on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command f81411ae on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4207 (Thu Sep 6 17:40:37 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4207 (Thu Sep 6 17:40:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2253735e", "local_conf_timestamp": 4207, "host-ts": 4207}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4203 (Thu Sep 6 17:40:33 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4203 (Thu Sep 6 17:40:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "27dd3283", "local_conf_timestamp": 4203, "host-ts": 4203}, "global_maintenance": false} lago.ssh: DEBUG: start task:e3708b7a-92ad-418c-812b-62b5a7848558:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:e3708b7a-92ad-418c-812b-62b5a7848558:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running fb99b770 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command fb99b770 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command fb99b770 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4207 (Thu Sep 6 17:40:37 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4207 (Thu Sep 6 17:40:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2253735e", "local_conf_timestamp": 4207, "host-ts": 4207}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4203 (Thu Sep 6 17:40:33 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4203 (Thu Sep 6 17:40:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "27dd3283", "local_conf_timestamp": 4203, "host-ts": 4203}, "global_maintenance": false} lago.ssh: DEBUG: start task:ac8d327c-bc33-4c15-81c4-091c7578adfc:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:ac8d327c-bc33-4c15-81c4-091c7578adfc:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running fe9fa182 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command fe9fa182 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command fe9fa182 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4217 (Thu Sep 6 17:40:48 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4217 (Thu Sep 6 17:40:48 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "8c604d9c", "local_conf_timestamp": 4217, "host-ts": 4217}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4213 (Thu Sep 6 17:40:43 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4213 (Thu Sep 6 17:40:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "239bcd86", "local_conf_timestamp": 4213, "host-ts": 4213}, "global_maintenance": false} lago.ssh: DEBUG: start task:6b6e4c48-4da2-4561-8e20-81c510232a9b:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:6b6e4c48-4da2-4561-8e20-81c510232a9b:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 018219fc on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 018219fc on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 018219fc on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4217 (Thu Sep 6 17:40:48 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4217 (Thu Sep 6 17:40:48 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "8c604d9c", "local_conf_timestamp": 4217, "host-ts": 4217}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4213 (Thu Sep 6 17:40:43 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4213 (Thu Sep 6 17:40:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "239bcd86", "local_conf_timestamp": 4213, "host-ts": 4213}, "global_maintenance": false} lago.ssh: DEBUG: start task:96227485-3980-4fdc-8b7a-6b8209beddba:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:96227485-3980-4fdc-8b7a-6b8209beddba:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 0464194a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 0464194a on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 0464194a on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4227 (Thu Sep 6 17:40:57 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4227 (Thu Sep 6 17:40:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "c32ada44", "local_conf_timestamp": 4227, "host-ts": 4227}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4223 (Thu Sep 6 17:40:53 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4224 (Thu Sep 6 17:40:54 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "b134592a", "local_conf_timestamp": 4224, "host-ts": 4223}, "global_maintenance": false} lago.ssh: DEBUG: start task:ff150e2b-4824-4415-bb85-e03c385d8164:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:ff150e2b-4824-4415-bb85-e03c385d8164:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 05762a44 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 05762a44 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 05762a44 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4227 (Thu Sep 6 17:40:57 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4227 (Thu Sep 6 17:40:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "c32ada44", "local_conf_timestamp": 4227, "host-ts": 4227}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4223 (Thu Sep 6 17:40:53 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4224 (Thu Sep 6 17:40:54 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "b134592a", "local_conf_timestamp": 4224, "host-ts": 4223}, "global_maintenance": false} lago.ssh: DEBUG: start task:277690ad-d660-43ec-91d6-219e1f44d4a3:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:277690ad-d660-43ec-91d6-219e1f44d4a3:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 0c8f00d0 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 0c8f00d0 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 0c8f00d0 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4237 (Thu Sep 6 17:41:07 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=4237 (Thu Sep 6 17:41:08 2018)\nconf_on_shared_storage=True\nmaintenance=True\nstate=LocalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 0, "stopped": false, "maintenance": true, "crc32": "055173ed", "local_conf_timestamp": 4237, "host-ts": 4237}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4233 (Thu Sep 6 17:41:03 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4233 (Thu Sep 6 17:41:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "86f105ab", "local_conf_timestamp": 4233, "host-ts": 4233}, "global_maintenance": false} root: INFO: * Engine has migrated. root: INFO: * Waiting For System Stability... cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 125, in local_maintenance _wait_for_engine_migration(host, he_index, "bad", "Migration Destination") File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 45, in _wait_for_engine_migration time.sleep(wait_value) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nlago.ssh: DEBUG: start task:dbac6aee-f807-447a-8c86-15134f2b7749:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: end task:dbac6aee-f807-447a-8c86-15134f2b7749:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: Running daa5b834 on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command daa5b834 on lago-he-node-ng-suite-4-2-host-1 returned with 0\nlago.ssh: DEBUG: Command daa5b834 on lago-he-node-ng-suite-4-2-host-1 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4146 (Thu Sep 6 17:39:36 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=4146 (Thu Sep 6 17:39:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e9066a29", "local_conf_timestamp": 4146, "host-ts": 4146}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4153 (Thu Sep 6 17:39:43 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4153 (Thu Sep 6 17:39:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "662a0a14", "local_conf_timestamp": 4153, "host-ts": 4153}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:d30f82be-086c-4823-8307-6871f22336f6:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:d30f82be-086c-4823-8307-6871f22336f6:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running dbb9947a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local\nlago.ssh: DEBUG: Command dbb9947a on lago-he-node-ng-suite-4-2-host-0 returned with 0\nroot: INFO: * Waiting for engine to migrate...\nlago.ssh: DEBUG: start task:7c4ae3ab-7501-4c92-9bea-fdaaef4859b2:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:7c4ae3ab-7501-4c92-9bea-fdaaef4859b2:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running dce9aff6 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command dce9aff6 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command dce9aff6 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4156 (Thu Sep 6 17:39:46 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=4156 (Thu Sep 6 17:39:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "35612a16", "local_conf_timestamp": 4156, "host-ts": 4156}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4153 (Thu Sep 6 17:39:43 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4153 (Thu Sep 6 17:39:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "662a0a14", "local_conf_timestamp": 4153, "host-ts": 4153}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:eb74d1b1-2d4e-416b-a20f-1bc0ce171837:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:eb74d1b1-2d4e-416b-a20f-1bc0ce171837:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running e4cbecf2 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e4cbecf2 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command e4cbecf2 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4177 (Thu Sep 6 17:40:07 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4177 (Thu Sep 6 17:40:08 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "40657674", "local_conf_timestamp": 4177, "host-ts": 4177}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4173 (Thu Sep 6 17:40:03 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4174 (Thu Sep 6 17:40:04 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f0c361bd", "local_conf_timestamp": 4174, "host-ts": 4173}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:9b2117bb-3e9b-4784-bb67-0d5f98f9d7b5:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:9b2117bb-3e9b-4784-bb67-0d5f98f9d7b5:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running e8606ab4 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e8606ab4 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command e8606ab4 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4177 (Thu Sep 6 17:40:07 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4177 (Thu Sep 6 17:40:08 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "40657674", "local_conf_timestamp": 4177, "host-ts": 4177}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4173 (Thu Sep 6 17:40:03 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4174 (Thu Sep 6 17:40:04 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f0c361bd", "local_conf_timestamp": 4174, "host-ts": 4173}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:ddbde6de-1c5d-42c4-9156-a1c592cb6dd0:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:ddbde6de-1c5d-42c4-9156-a1c592cb6dd0:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running eb997522 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command eb997522 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command eb997522 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4187 (Thu Sep 6 17:40:17 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4187 (Thu Sep 6 17:40:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "cdd0276f", "local_conf_timestamp": 4187, "host-ts": 4187}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4183 (Thu Sep 6 17:40:13 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4183 (Thu Sep 6 17:40:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "267dc4b5", "local_conf_timestamp": 4183, "host-ts": 4183}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:fc0e0ad5-af11-42b1-8b1a-542db357c10f:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:fc0e0ad5-af11-42b1-8b1a-542db357c10f:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running ee95607e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ee95607e on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command ee95607e on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4187 (Thu Sep 6 17:40:17 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4187 (Thu Sep 6 17:40:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "cdd0276f", "local_conf_timestamp": 4187, "host-ts": 4187}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4183 (Thu Sep 6 17:40:13 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4183 (Thu Sep 6 17:40:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "267dc4b5", "local_conf_timestamp": 4183, "host-ts": 4183}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:33c75a20-69a6-4f1f-9d10-5964b70e5476:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:33c75a20-69a6-4f1f-9d10-5964b70e5476:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running f22070d0 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f22070d0 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command f22070d0 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4197 (Thu Sep 6 17:40:27 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4197 (Thu Sep 6 17:40:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "63e319ad", "local_conf_timestamp": 4197, "host-ts": 4197}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4193 (Thu Sep 6 17:40:23 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4193 (Thu Sep 6 17:40:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "223b3bb0", "local_conf_timestamp": 4193, "host-ts": 4193}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:eebbdb9b-3cca-497b-83fb-90f5b365bd00:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:eebbdb9b-3cca-497b-83fb-90f5b365bd00:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running f5324a00 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f5324a00 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command f5324a00 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4197 (Thu Sep 6 17:40:27 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4197 (Thu Sep 6 17:40:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "63e319ad", "local_conf_timestamp": 4197, "host-ts": 4197}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4193 (Thu Sep 6 17:40:23 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4193 (Thu Sep 6 17:40:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "223b3bb0", "local_conf_timestamp": 4193, "host-ts": 4193}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:901e9f22-e329-4658-a951-0e1aa80b3cd6:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:901e9f22-e329-4658-a951-0e1aa80b3cd6:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running f81411ae on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f81411ae on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command f81411ae on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4207 (Thu Sep 6 17:40:37 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4207 (Thu Sep 6 17:40:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2253735e", "local_conf_timestamp": 4207, "host-ts": 4207}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4203 (Thu Sep 6 17:40:33 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4203 (Thu Sep 6 17:40:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "27dd3283", "local_conf_timestamp": 4203, "host-ts": 4203}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:e3708b7a-92ad-418c-812b-62b5a7848558:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:e3708b7a-92ad-418c-812b-62b5a7848558:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running fb99b770 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fb99b770 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command fb99b770 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4207 (Thu Sep 6 17:40:37 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4207 (Thu Sep 6 17:40:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2253735e", "local_conf_timestamp": 4207, "host-ts": 4207}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4203 (Thu Sep 6 17:40:33 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4203 (Thu Sep 6 17:40:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "27dd3283", "local_conf_timestamp": 4203, "host-ts": 4203}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:ac8d327c-bc33-4c15-81c4-091c7578adfc:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:ac8d327c-bc33-4c15-81c4-091c7578adfc:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running fe9fa182 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fe9fa182 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command fe9fa182 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4217 (Thu Sep 6 17:40:48 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4217 (Thu Sep 6 17:40:48 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "8c604d9c", "local_conf_timestamp": 4217, "host-ts": 4217}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4213 (Thu Sep 6 17:40:43 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4213 (Thu Sep 6 17:40:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "239bcd86", "local_conf_timestamp": 4213, "host-ts": 4213}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:6b6e4c48-4da2-4561-8e20-81c510232a9b:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:6b6e4c48-4da2-4561-8e20-81c510232a9b:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 018219fc on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 018219fc on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 018219fc on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4217 (Thu Sep 6 17:40:48 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4217 (Thu Sep 6 17:40:48 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "8c604d9c", "local_conf_timestamp": 4217, "host-ts": 4217}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4213 (Thu Sep 6 17:40:43 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4213 (Thu Sep 6 17:40:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "239bcd86", "local_conf_timestamp": 4213, "host-ts": 4213}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:96227485-3980-4fdc-8b7a-6b8209beddba:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:96227485-3980-4fdc-8b7a-6b8209beddba:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 0464194a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 0464194a on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 0464194a on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4227 (Thu Sep 6 17:40:57 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4227 (Thu Sep 6 17:40:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "c32ada44", "local_conf_timestamp": 4227, "host-ts": 4227}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4223 (Thu Sep 6 17:40:53 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4224 (Thu Sep 6 17:40:54 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "b134592a", "local_conf_timestamp": 4224, "host-ts": 4223}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:ff150e2b-4824-4415-bb85-e03c385d8164:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:ff150e2b-4824-4415-bb85-e03c385d8164:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 05762a44 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 05762a44 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 05762a44 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4227 (Thu Sep 6 17:40:57 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4227 (Thu Sep 6 17:40:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "c32ada44", "local_conf_timestamp": 4227, "host-ts": 4227}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4223 (Thu Sep 6 17:40:53 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4224 (Thu Sep 6 17:40:54 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "b134592a", "local_conf_timestamp": 4224, "host-ts": 4223}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:277690ad-d660-43ec-91d6-219e1f44d4a3:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:277690ad-d660-43ec-91d6-219e1f44d4a3:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 0c8f00d0 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 0c8f00d0 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 0c8f00d0 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4237 (Thu Sep 6 17:41:07 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=4237 (Thu Sep 6 17:41:08 2018)\\nconf_on_shared_storage=True\\nmaintenance=True\\nstate=LocalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 0, "stopped": false, "maintenance": true, "crc32": "055173ed", "local_conf_timestamp": 4237, "host-ts": 4237}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4233 (Thu Sep 6 17:41:03 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4233 (Thu Sep 6 17:41:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "86f105ab", "local_conf_timestamp": 4233, "host-ts": 4233}, "global_maintenance": false}\n\nroot: INFO: * Engine has migrated.\nroot: INFO: * Waiting For System Stability...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'