[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-4.2 - Build # 409 - Still Failing!

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/409/ Build Number: 409 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #401 [Eitan Raviv] networking: connect NFS to engine over IPv6 Changes for Build #402 [Eitan Raviv] networking: connect NFS to engine over IPv6 Changes for Build #403 [Eitan Raviv] networking: connect NFS to engine over IPv6 Changes for Build #404 [Gal Ben Haim] Add run-ost-container Changes for Build #405 [Gal Ben Haim] Add run-ost-container Changes for Build #406 [Gal Ben Haim] Add run-ost-container [Evgheni Dereveanchin] Remove fcraw mirror sync job Changes for Build #407 [Gal Ben Haim] Add run-ost-container [Dafna Ron] Removed ovirt-provider-ovn v1 jobs Changes for Build #408 [Gal Ben Haim] OST container improvements Changes for Build #409 [Gal Ben Haim] OST container improvements [Sandro Bonazzola] Revert "Removed ovirt-provider-ovn v1 jobs" ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 010_local_maintenance_cli.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... lago.ssh: DEBUG: start task:057752aa-35e7-4b85-b5ef-addebc75c5b3:Get ssh client for lago-he-node-ng-suite-4-2-host-1: lago.ssh: DEBUG: end task:057752aa-35e7-4b85-b5ef-addebc75c5b3:Get ssh client for lago-he-node-ng-suite-4-2-host-1: lago.ssh: DEBUG: Running 87d3da2c on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 87d3da2c on lago-he-node-ng-suite-4-2-host-1 returned with 0 lago.ssh: DEBUG: Command 87d3da2c on lago-he-node-ng-suite-4-2-host-1 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3881 (Wed Oct 17 09:39:19 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=3881 (Wed Oct 17 09:39:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "12c7a981", "local_conf_timestamp": 3881, "host-ts": 3881}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3876 (Wed Oct 17 09:39:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3876 (Wed Oct 17 09:39:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c64ad9da", "local_conf_timestamp": 3876, "host-ts": 3876}, "global_maintenance": false} lago.ssh: DEBUG: start task:f31e6bd1-19e2-46c0-9b01-58be37a5b1b5:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:f31e6bd1-19e2-46c0-9b01-58be37a5b1b5:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 88ec85d0 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local lago.ssh: DEBUG: Command 88ec85d0 on lago-he-node-ng-suite-4-2-host-0 returned with 0 root: INFO: * Waiting for engine to migrate... lago.ssh: DEBUG: start task:a3bee47a-7f24-45c7-afda-b492dcb12b7f:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:a3bee47a-7f24-45c7-afda-b492dcb12b7f:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 8a074248 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 8a074248 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 8a074248 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3881 (Wed Oct 17 09:39:19 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=3881 (Wed Oct 17 09:39:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "12c7a981", "local_conf_timestamp": 3881, "host-ts": 3881}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3886 (Wed Oct 17 09:39:24 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3886 (Wed Oct 17 09:39:24 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f9addce9", "local_conf_timestamp": 3886, "host-ts": 3886}, "global_maintenance": false} lago.ssh: DEBUG: start task:d99ed652-3376-477e-ab81-cf00dcdb210b:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:d99ed652-3376-477e-ab81-cf00dcdb210b:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 91115038 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 91115038 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 91115038 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3901 (Wed Oct 17 09:39:39 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3901 (Wed Oct 17 09:39:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2ff36035", "local_conf_timestamp": 3901, "host-ts": 3901}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3896 (Wed Oct 17 09:39:34 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3896 (Wed Oct 17 09:39:34 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fdeb23ec", "local_conf_timestamp": 3896, "host-ts": 3896}, "global_maintenance": false} lago.ssh: DEBUG: start task:d320697f-f885-4783-817c-b2fa75a64385:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:d320697f-f885-4783-817c-b2fa75a64385:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 9446201c on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9446201c on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 9446201c on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3901 (Wed Oct 17 09:39:39 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3901 (Wed Oct 17 09:39:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2ff36035", "local_conf_timestamp": 3901, "host-ts": 3901}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3896 (Wed Oct 17 09:39:34 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3896 (Wed Oct 17 09:39:34 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fdeb23ec", "local_conf_timestamp": 3896, "host-ts": 3896}, "global_maintenance": false} lago.ssh: DEBUG: start task:5c0be3ed-f852-4fc8-ba22-79dc9ddc6fd9:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:5c0be3ed-f852-4fc8-ba22-79dc9ddc6fd9:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 9733e5e8 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9733e5e8 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 9733e5e8 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3911 (Wed Oct 17 09:39:49 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3911 (Wed Oct 17 09:39:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "81c05ef7", "local_conf_timestamp": 3911, "host-ts": 3911}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3906 (Wed Oct 17 09:39:45 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3906 (Wed Oct 17 09:39:45 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "dcc9a329", "local_conf_timestamp": 3906, "host-ts": 3906}, "global_maintenance": false} lago.ssh: DEBUG: start task:572cca72-4ed1-43d2-ac3f-360688d6e85c:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:572cca72-4ed1-43d2-ac3f-360688d6e85c:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 9a1dc79c on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9a1dc79c on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 9a1dc79c on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3911 (Wed Oct 17 09:39:49 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3911 (Wed Oct 17 09:39:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "81c05ef7", "local_conf_timestamp": 3911, "host-ts": 3911}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3906 (Wed Oct 17 09:39:45 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3906 (Wed Oct 17 09:39:45 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "dcc9a329", "local_conf_timestamp": 3906, "host-ts": 3906}, "global_maintenance": false} lago.ssh: DEBUG: start task:b424ae52-a6e2-4503-9307-ab80af7c11d7:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:b424ae52-a6e2-4503-9307-ab80af7c11d7:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 9d0dbf70 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9d0dbf70 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 9d0dbf70 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3911 (Wed Oct 17 09:39:49 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3911 (Wed Oct 17 09:39:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "81c05ef7", "local_conf_timestamp": 3911, "host-ts": 3911}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3916 (Wed Oct 17 09:39:54 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3916 (Wed Oct 17 09:39:54 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d88f5c2c", "local_conf_timestamp": 3916, "host-ts": 3916}, "global_maintenance": false} lago.ssh: DEBUG: start task:8c50aed0-f059-41d9-afa6-7f059dd76007:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:8c50aed0-f059-41d9-afa6-7f059dd76007:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running a026a8f2 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a026a8f2 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command a026a8f2 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3921 (Wed Oct 17 09:40:00 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3922 (Wed Oct 17 09:40:00 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "31ed4a4a", "local_conf_timestamp": 3922, "host-ts": 3921}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3916 (Wed Oct 17 09:39:54 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3916 (Wed Oct 17 09:39:54 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d88f5c2c", "local_conf_timestamp": 3916, "host-ts": 3916}, "global_maintenance": false} lago.ssh: DEBUG: start task:bf7a8136-be10-4023-b9cc-fa6837346dee:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:bf7a8136-be10-4023-b9cc-fa6837346dee:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running a33cd192 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a33cd192 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command a33cd192 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3931 (Wed Oct 17 09:40:09 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3931 (Wed Oct 17 09:40:10 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "06d72532", "local_conf_timestamp": 3931, "host-ts": 3931}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3926 (Wed Oct 17 09:40:04 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3926 (Wed Oct 17 09:40:04 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "676085ad", "local_conf_timestamp": 3926, "host-ts": 3926}, "global_maintenance": false} lago.ssh: DEBUG: start task:82b629e2-09f4-4321-9c55-8afbb15910a6:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:82b629e2-09f4-4321-9c55-8afbb15910a6:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running a6a01934 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a6a01934 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command a6a01934 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3931 (Wed Oct 17 09:40:09 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3931 (Wed Oct 17 09:40:10 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "06d72532", "local_conf_timestamp": 3931, "host-ts": 3931}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3926 (Wed Oct 17 09:40:04 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3926 (Wed Oct 17 09:40:04 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "676085ad", "local_conf_timestamp": 3926, "host-ts": 3926}, "global_maintenance": false} lago.ssh: DEBUG: start task:783ed088-c50c-40da-b9a5-0d81e269d484:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:783ed088-c50c-40da-b9a5-0d81e269d484:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running a9aa9082 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a9aa9082 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command a9aa9082 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3941 (Wed Oct 17 09:40:20 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3942 (Wed Oct 17 09:40:20 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "63a5c044", "local_conf_timestamp": 3942, "host-ts": 3941}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3936 (Wed Oct 17 09:40:14 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3936 (Wed Oct 17 09:40:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "63267aa8", "local_conf_timestamp": 3936, "host-ts": 3936}, "global_maintenance": false} lago.ssh: DEBUG: start task:a8d4bb44-0407-4ae5-b86c-e7b6ee37adc7:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:a8d4bb44-0407-4ae5-b86c-e7b6ee37adc7:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running acb2c646 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command acb2c646 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command acb2c646 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3941 (Wed Oct 17 09:40:20 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3942 (Wed Oct 17 09:40:20 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "63a5c044", "local_conf_timestamp": 3942, "host-ts": 3941}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3936 (Wed Oct 17 09:40:14 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3936 (Wed Oct 17 09:40:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "63267aa8", "local_conf_timestamp": 3936, "host-ts": 3936}, "global_maintenance": false} lago.ssh: DEBUG: start task:930e4fe6-a51d-4ed5-9a2b-0d30c1406325:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:930e4fe6-a51d-4ed5-9a2b-0d30c1406325:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running afac868e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command afac868e on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command afac868e on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3951 (Wed Oct 17 09:40:29 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3951 (Wed Oct 17 09:40:29 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "VM migrated away successfully", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ccb49254", "local_conf_timestamp": 3951, "host-ts": 3951}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3946 (Wed Oct 17 09:40:25 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3946 (Wed Oct 17 09:40:25 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7ef687b3", "local_conf_timestamp": 3946, "host-ts": 3946}, "global_maintenance": false} lago.ssh: DEBUG: start task:fff4cd78-4fd2-4bfa-99bf-cd4e3858100a:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:fff4cd78-4fd2-4bfa-99bf-cd4e3858100a:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running b0ee892a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b0ee892a on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command b0ee892a on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3951 (Wed Oct 17 09:40:29 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3951 (Wed Oct 17 09:40:29 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "VM migrated away successfully", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ccb49254", "local_conf_timestamp": 3951, "host-ts": 3951}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3946 (Wed Oct 17 09:40:25 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3946 (Wed Oct 17 09:40:25 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7ef687b3", "local_conf_timestamp": 3946, "host-ts": 3946}, "global_maintenance": false} lago.ssh: DEBUG: start task:9ae51024-a39a-44a9-b913-8f2c4d5bb0c5:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:9ae51024-a39a-44a9-b913-8f2c4d5bb0c5:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running b808ddaa on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b808ddaa on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command b808ddaa on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3961 (Wed Oct 17 09:40:40 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=3961 (Wed Oct 17 09:40:40 2018)\nconf_on_shared_storage=True\nmaintenance=True\nstate=LocalMaintenance\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 0, "stopped": false, "maintenance": true, "crc32": "4b55e0ce", "local_conf_timestamp": 3961, "host-ts": 3961}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3956 (Wed Oct 17 09:40:34 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3956 (Wed Oct 17 09:40:34 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "44ef2a27", "local_conf_timestamp": 3956, "host-ts": 3956}, "global_maintenance": false} root: INFO: * Engine has migrated. root: INFO: * Waiting For System Stability... cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 125, in local_maintenance _wait_for_engine_migration(host, he_index, "bad", "Migration Destination") File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 45, in _wait_for_engine_migration time.sleep(wait_value) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nlago.ssh: DEBUG: start task:057752aa-35e7-4b85-b5ef-addebc75c5b3:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: end task:057752aa-35e7-4b85-b5ef-addebc75c5b3:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: Running 87d3da2c on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 87d3da2c on lago-he-node-ng-suite-4-2-host-1 returned with 0\nlago.ssh: DEBUG: Command 87d3da2c on lago-he-node-ng-suite-4-2-host-1 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3881 (Wed Oct 17 09:39:19 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=3881 (Wed Oct 17 09:39:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "12c7a981", "local_conf_timestamp": 3881, "host-ts": 3881}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3876 (Wed Oct 17 09:39:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3876 (Wed Oct 17 09:39:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c64ad9da", "local_conf_timestamp": 3876, "host-ts": 3876}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:f31e6bd1-19e2-46c0-9b01-58be37a5b1b5:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:f31e6bd1-19e2-46c0-9b01-58be37a5b1b5:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 88ec85d0 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local\nlago.ssh: DEBUG: Command 88ec85d0 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nroot: INFO: * Waiting for engine to migrate...\nlago.ssh: DEBUG: start task:a3bee47a-7f24-45c7-afda-b492dcb12b7f:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:a3bee47a-7f24-45c7-afda-b492dcb12b7f:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 8a074248 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 8a074248 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 8a074248 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3881 (Wed Oct 17 09:39:19 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=3881 (Wed Oct 17 09:39:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "12c7a981", "local_conf_timestamp": 3881, "host-ts": 3881}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3886 (Wed Oct 17 09:39:24 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3886 (Wed Oct 17 09:39:24 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f9addce9", "local_conf_timestamp": 3886, "host-ts": 3886}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:d99ed652-3376-477e-ab81-cf00dcdb210b:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:d99ed652-3376-477e-ab81-cf00dcdb210b:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 91115038 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 91115038 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 91115038 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3901 (Wed Oct 17 09:39:39 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3901 (Wed Oct 17 09:39:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2ff36035", "local_conf_timestamp": 3901, "host-ts": 3901}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3896 (Wed Oct 17 09:39:34 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3896 (Wed Oct 17 09:39:34 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fdeb23ec", "local_conf_timestamp": 3896, "host-ts": 3896}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:d320697f-f885-4783-817c-b2fa75a64385:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:d320697f-f885-4783-817c-b2fa75a64385:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 9446201c on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9446201c on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 9446201c on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3901 (Wed Oct 17 09:39:39 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3901 (Wed Oct 17 09:39:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2ff36035", "local_conf_timestamp": 3901, "host-ts": 3901}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3896 (Wed Oct 17 09:39:34 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3896 (Wed Oct 17 09:39:34 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fdeb23ec", "local_conf_timestamp": 3896, "host-ts": 3896}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:5c0be3ed-f852-4fc8-ba22-79dc9ddc6fd9:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:5c0be3ed-f852-4fc8-ba22-79dc9ddc6fd9:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 9733e5e8 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9733e5e8 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 9733e5e8 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3911 (Wed Oct 17 09:39:49 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3911 (Wed Oct 17 09:39:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "81c05ef7", "local_conf_timestamp": 3911, "host-ts": 3911}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3906 (Wed Oct 17 09:39:45 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3906 (Wed Oct 17 09:39:45 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "dcc9a329", "local_conf_timestamp": 3906, "host-ts": 3906}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:572cca72-4ed1-43d2-ac3f-360688d6e85c:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:572cca72-4ed1-43d2-ac3f-360688d6e85c:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 9a1dc79c on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9a1dc79c on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 9a1dc79c on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3911 (Wed Oct 17 09:39:49 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3911 (Wed Oct 17 09:39:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "81c05ef7", "local_conf_timestamp": 3911, "host-ts": 3911}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3906 (Wed Oct 17 09:39:45 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3906 (Wed Oct 17 09:39:45 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "dcc9a329", "local_conf_timestamp": 3906, "host-ts": 3906}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:b424ae52-a6e2-4503-9307-ab80af7c11d7:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:b424ae52-a6e2-4503-9307-ab80af7c11d7:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 9d0dbf70 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9d0dbf70 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 9d0dbf70 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3911 (Wed Oct 17 09:39:49 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3911 (Wed Oct 17 09:39:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "81c05ef7", "local_conf_timestamp": 3911, "host-ts": 3911}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3916 (Wed Oct 17 09:39:54 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3916 (Wed Oct 17 09:39:54 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d88f5c2c", "local_conf_timestamp": 3916, "host-ts": 3916}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:8c50aed0-f059-41d9-afa6-7f059dd76007:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:8c50aed0-f059-41d9-afa6-7f059dd76007:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running a026a8f2 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a026a8f2 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command a026a8f2 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3921 (Wed Oct 17 09:40:00 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3922 (Wed Oct 17 09:40:00 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "31ed4a4a", "local_conf_timestamp": 3922, "host-ts": 3921}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3916 (Wed Oct 17 09:39:54 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3916 (Wed Oct 17 09:39:54 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d88f5c2c", "local_conf_timestamp": 3916, "host-ts": 3916}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:bf7a8136-be10-4023-b9cc-fa6837346dee:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:bf7a8136-be10-4023-b9cc-fa6837346dee:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running a33cd192 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a33cd192 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command a33cd192 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3931 (Wed Oct 17 09:40:09 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3931 (Wed Oct 17 09:40:10 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "06d72532", "local_conf_timestamp": 3931, "host-ts": 3931}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3926 (Wed Oct 17 09:40:04 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3926 (Wed Oct 17 09:40:04 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "676085ad", "local_conf_timestamp": 3926, "host-ts": 3926}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:82b629e2-09f4-4321-9c55-8afbb15910a6:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:82b629e2-09f4-4321-9c55-8afbb15910a6:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running a6a01934 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a6a01934 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command a6a01934 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3931 (Wed Oct 17 09:40:09 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3931 (Wed Oct 17 09:40:10 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "06d72532", "local_conf_timestamp": 3931, "host-ts": 3931}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3926 (Wed Oct 17 09:40:04 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3926 (Wed Oct 17 09:40:04 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "676085ad", "local_conf_timestamp": 3926, "host-ts": 3926}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:783ed088-c50c-40da-b9a5-0d81e269d484:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:783ed088-c50c-40da-b9a5-0d81e269d484:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running a9aa9082 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a9aa9082 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command a9aa9082 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3941 (Wed Oct 17 09:40:20 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3942 (Wed Oct 17 09:40:20 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "63a5c044", "local_conf_timestamp": 3942, "host-ts": 3941}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3936 (Wed Oct 17 09:40:14 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3936 (Wed Oct 17 09:40:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "63267aa8", "local_conf_timestamp": 3936, "host-ts": 3936}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:a8d4bb44-0407-4ae5-b86c-e7b6ee37adc7:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:a8d4bb44-0407-4ae5-b86c-e7b6ee37adc7:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running acb2c646 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command acb2c646 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command acb2c646 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3941 (Wed Oct 17 09:40:20 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3942 (Wed Oct 17 09:40:20 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "63a5c044", "local_conf_timestamp": 3942, "host-ts": 3941}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3936 (Wed Oct 17 09:40:14 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3936 (Wed Oct 17 09:40:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "63267aa8", "local_conf_timestamp": 3936, "host-ts": 3936}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:930e4fe6-a51d-4ed5-9a2b-0d30c1406325:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:930e4fe6-a51d-4ed5-9a2b-0d30c1406325:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running afac868e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command afac868e on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command afac868e on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3951 (Wed Oct 17 09:40:29 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3951 (Wed Oct 17 09:40:29 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "VM migrated away successfully", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ccb49254", "local_conf_timestamp": 3951, "host-ts": 3951}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3946 (Wed Oct 17 09:40:25 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3946 (Wed Oct 17 09:40:25 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7ef687b3", "local_conf_timestamp": 3946, "host-ts": 3946}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:fff4cd78-4fd2-4bfa-99bf-cd4e3858100a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:fff4cd78-4fd2-4bfa-99bf-cd4e3858100a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b0ee892a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b0ee892a on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command b0ee892a on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3951 (Wed Oct 17 09:40:29 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3951 (Wed Oct 17 09:40:29 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "VM migrated away successfully", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ccb49254", "local_conf_timestamp": 3951, "host-ts": 3951}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3946 (Wed Oct 17 09:40:25 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3946 (Wed Oct 17 09:40:25 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7ef687b3", "local_conf_timestamp": 3946, "host-ts": 3946}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:9ae51024-a39a-44a9-b913-8f2c4d5bb0c5:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:9ae51024-a39a-44a9-b913-8f2c4d5bb0c5:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b808ddaa on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b808ddaa on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command b808ddaa on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3961 (Wed Oct 17 09:40:40 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=3961 (Wed Oct 17 09:40:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=True\\nstate=LocalMaintenance\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 0, "stopped": false, "maintenance": true, "crc32": "4b55e0ce", "local_conf_timestamp": 3961, "host-ts": 3961}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3956 (Wed Oct 17 09:40:34 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3956 (Wed Oct 17 09:40:34 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "44ef2a27", "local_conf_timestamp": 3956, "host-ts": 3956}, "global_maintenance": false}\n\nroot: INFO: * Engine has migrated.\nroot: INFO: * Waiting For System Stability...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'