[JIRA] (OVIRT-2499) Add support for tag stage in STDCI
by Daniel Belenky (oVirt JIRA)
Daniel Belenky created OVIRT-2499:
-------------------------------------
Summary: Add support for tag stage in STDCI
Key: OVIRT-2499
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2499
Project: oVirt - virtualization made easy
Issue Type: Epic
Components: Standard CI (Pipelines), STDCI DSL
Reporter: Daniel Belenky
Assignee: infra
As a developer, I want to have the ability to run my automation code upon new tags being pushed to my project so that I can trigger an automated release process.
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100092)
6 years, 2 months
[JIRA] (OVIRT-2498) Failing KubeVirt CI
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2498?page=com.atlassian.jir... ]
Barak Korren commented on OVIRT-2498:
-------------------------------------
[~pkotas] I think the best place to discuss Kubevirt issues is on the Kubevirt-related mailing lists where other Kubvirt developers can see the discussion.
To your questions:
{quote}
I am working on fixing the issues on the KubeVirt e2e test suites. This
task is directly related to unstable CI, due to unknown errors.
The progress is reported in the CNV trello:
https://trello.com/c/HNXcMEQu/161-epic-improve-ci
I am creating this issue since the KubeVirt experience random timeouts on
random tests most of the times when test suites run.
The issue from outside is showing as timeouts on difference part of tests.
Sometimes the tests fails in set up phase, again due to random timeout.
The example in the link bellow timed out for network connection on
localhost.
[check-patch.k8s-1.11.0-dev.el7.x86_64]
requests.exceptions.ReadTimeout:
UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.
(read timeout=60)
{quote}
Its generally a bad idea to rely to much on timeout in a test suit like this. We've seen such issues over and over again in OST as well. Its probably best to remove all such timeout definitions and just have overall timeout set for the entire test suit.
{quote}
Example of failing test suites is here
https://jenkins.ovirt.org/job/kubevirt_kubevirt_standard-check-pr/1916/co...
The list of errors related to the failing CI can be found in my notes
https://docs.google.com/document/d/1_ll1DOMHgCRHn_Df9i4uvtRFyMK-bDCHEeGfJ...
I am not sure whether KubeVirt already shared the resource requirements, so
I provide short summary:
Resources for KubeVirt e2e tests:
at least 12GB of RAM - we start 3 nodes (3 docker images) each require
4GB of RAM
exposed /dev/kvm to enable native virtualization
cached images, since these are used to build the test cluster:
kubevirtci/os-3.10.0-crio:latest
kubevirtci/os-3.10.0-multus:latest
kubevirtci/os-3.10.0:latest
kubevirtci/k8s-1.10.4:latest
kubevirtci/k8s-multus-1.11.1:latest
kubevirtci/k8s-1.11.0:latest
How can we overcome this? Can we work together to build a suitable
requirements for running the tests so it passes each time?
{quote}
To my knowledge the existing setup meets all the requirements you specify above.
We have 3 physical hosts that are used to run Kubevirt tests, each host has 128GB of ram and runs 7 containers where each container runs its own Libvirt, Docker and Systemd so that it looks like its own host to the tests running inside. The amount of containers per host was calculated to have each container have a little over 16GB of RAM for itself. So we should have more then enough for Kubevirt. Also, in our measurements Kubevirt's CI tests took way less then 12GB, ansd were somewhere around 8GB.
All the images that start with 'kubevirtci' are cached by the system.
WRT /dev/kvm - we do have it exposed in the containers we run, but I think that is irrelevant since AFAIK Kubevirt-CI runs qemu on its own inside its own container, so the /dev/kvm device files simply needs to exist inside that container.
> Failing KubeVirt CI
> -------------------
>
> Key: OVIRT-2498
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2498
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: Petr Kotas
> Assignee: infra
>
> Hi,
> I am working on fixing the issues on the KubeVirt e2e test suites. This
> task is directly related to unstable CI, due to unknown errors.
> The progress is reported in the CNV trello:
> https://trello.com/c/HNXcMEQu/161-epic-improve-ci
> I am creating this issue since the KubeVirt experience random timeouts on
> random tests most of the times when test suites run.
> The issue from outside is showing as timeouts on difference part of tests.
> Sometimes the tests fails in set up phase, again due to random timeout.
> The example in the link bellow timed out for network connection on
> localhost.
> [check-patch.k8s-1.11.0-dev.el7.x86_64]
> requests.exceptions.ReadTimeout:
> UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.
> (read timeout=60)
> Example of failing test suites is here
> https://jenkins.ovirt.org/job/kubevirt_kubevirt_standard-check-pr/1916/co...
> The list of errors related to the failing CI can be found in my notes
> https://docs.google.com/document/d/1_ll1DOMHgCRHn_Df9i4uvtRFyMK-bDCHEeGfJ...
> I am not sure whether KubeVirt already shared the resource requirements, so
> I provide short summary:
> *Resources for KubeVirt e2e tests:*
> - at least 12GB of RAM - we start 3 nodes (3 docker images) each require
> 4GB of RAM
> - exposed /dev/kvm to enable native virtualization
> - cached images, since these are used to build the test cluster:
> - kubevirtci/os-3.10.0-crio:latest
> - kubevirtci/os-3.10.0-multus:latest
> - kubevirtci/os-3.10.0:latest
> - kubevirtci/k8s-1.10.4:latest
> - kubevirtci/k8s-multus-1.11.1:latest
> - kubevirtci/k8s-1.11.0:latest
> How can we overcome this? Can we work together to build a suitable
> requirements for running the tests so it passes each time?
> Kind regards,
> Petr Kotas
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100092)
6 years, 2 months
[oVirt Jenkins] ovirt-system-tests_he-basic-ansible-suite-master -
Build # 632 - Failure!
by jenkins@jenkins.phx.ovirt.org
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-ma...
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-ma...
Build Number: 632
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #632
[Milan Zamazal] Update test-vm.ovf file
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: 008_restart_he_vm.restart_he_vm
Error Message:
could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2561 (Sat Sep 15 02:53:03 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2561 (Sat Sep 15 02:53:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2e14c6de", "local_conf_timestamp": 2561, "host-ts": 2561}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2552 (Sat Sep 15 02:52:54 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2552 (Sat Sep 15 02:52:54 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-
-------------------- >> begin captured logging << --------------------
lago.ssh: DEBUG: start task:ed47f4dc-58b7-4cd6-8718-820f742fbaa8:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:ed47f4dc-58b7-4cd6-8718-820f742fbaa8:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running 9face572 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 9face572 on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 9face572 on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2397 (Sat Sep 15 02:50:19 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2397 (Sat Sep 15 02:50:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5f097b86", "local_conf_timestamp": 2397, "host-ts": 2397}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2392 (Sat Sep 15 02:50:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2392 (Sat Sep 15 02:50:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "524cc788", "local_conf_timestamp": 2392, "host-ts": 2392}, "global_maintenance": true}
root: INFO: * Shutting down HE VM on host: lago-he-basic-ansible-suite-master-host-0
lago.ssh: DEBUG: start task:9d2bbc93-b674-4f8f-9cff-0ed017480579:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:9d2bbc93-b674-4f8f-9cff-0ed017480579:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running a03b5ad2 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-shutdown
lago.ssh: DEBUG: Command a03b5ad2 on lago-he-basic-ansible-suite-master-host-0 returned with 0
root: INFO: * Command succeeded
root: INFO: * Waiting for VM to be down...
lago.ssh: DEBUG: start task:c6f14484-bc15-46ae-a11a-d71e7d7736e6:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:c6f14484-bc15-46ae-a11a-d71e7d7736e6:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running a0d7f720 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command a0d7f720 on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command a0d7f720 on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2397 (Sat Sep 15 02:50:19 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2397 (Sat Sep 15 02:50:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5f097b86", "local_conf_timestamp": 2397, "host-ts": 2397}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2392 (Sat Sep 15 02:50:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2392 (Sat Sep 15 02:50:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "524cc788", "local_conf_timestamp": 2392, "host-ts": 2392}, "global_maintenance": true}
lago.ssh: DEBUG: start task:b7f4b056-1ffd-4395-86d7-cbdba57b299a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:b7f4b056-1ffd-4395-86d7-cbdba57b299a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running a76cfd56 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command a76cfd56 on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command a76cfd56 on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2407 (Sat Sep 15 02:50:29 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2407 (Sat Sep 15 02:50:29 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f828e9c5", "local_conf_timestamp": 2407, "host-ts": 2407}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2402 (Sat Sep 15 02:50:24 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2402 (Sat Sep 15 02:50:25 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d36639d7", "local_conf_timestamp": 2402, "host-ts": 2402}, "global_maintenance": true}
lago.ssh: DEBUG: start task:0be59cd9-e4cb-4839-9891-894d2a5f611e:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:0be59cd9-e4cb-4839-9891-894d2a5f611e:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running a9bc073c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command a9bc073c on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command a9bc073c on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2417 (Sat Sep 15 02:50:39 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2417 (Sat Sep 15 02:50:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "42b65cd2", "local_conf_timestamp": 2417, "host-ts": 2417}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2412 (Sat Sep 15 02:50:34 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2412 (Sat Sep 15 02:50:34 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "db6596aa", "local_conf_timestamp": 2412, "host-ts": 2412}, "global_maintenance": true}
root: INFO: * VM is down.
root: INFO: * Stopping services...
lago.ssh: DEBUG: start task:a7f11961-66fe-4a3e-8b5c-03640709deb3:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:a7f11961-66fe-4a3e-8b5c-03640709deb3:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running aa528b4e on lago-he-basic-ansible-suite-master-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent
lago.ssh: DEBUG: Command aa528b4e on lago-he-basic-ansible-suite-master-host-0 returned with 0
root: INFO: * Starting services...
lago.ssh: DEBUG: start task:3470e088-3896-444c-9480-28f84251a111:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:3470e088-3896-444c-9480-28f84251a111:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running b42c375a on lago-he-basic-ansible-suite-master-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent
lago.ssh: DEBUG: Command b42c375a on lago-he-basic-ansible-suite-master-host-0 returned with 0
root: INFO: * Waiting for agent to be ready...
lago.ssh: DEBUG: start task:c551d6c2-0baf-48c5-b907-f31f4391d1c6:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:c551d6c2-0baf-48c5-b907-f31f4391d1c6:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running b677b14c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command b677b14c on lago-he-basic-ansible-suite-master-host-0 returned with 1
lago.ssh: DEBUG: Command b677b14c on lago-he-basic-ansible-suite-master-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:5dfa64c5-57dd-43f8-b09b-5e6549669059:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:5dfa64c5-57dd-43f8-b09b-5e6549669059:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running bcc603e6 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command bcc603e6 on lago-he-basic-ansible-suite-master-host-0 returned with 1
lago.ssh: DEBUG: Command bcc603e6 on lago-he-basic-ansible-suite-master-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:4100a51d-ba55-4808-ac40-f68b20c91e05:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:4100a51d-ba55-4808-ac40-f68b20c91e05:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running beff4df2 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command beff4df2 on lago-he-basic-ansible-suite-master-host-0 returned with 1
lago.ssh: DEBUG: Command beff4df2 on lago-he-basic-ansible-suite-master-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:46fa8f10-bbc9-433e-9b53-0029800375c7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:46fa8f10-bbc9-433e-9b53-0029800375c7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running c127b556 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command c127b556 on lago-he-basic-ansible-suite-master-host-0 returned with 1
lago.ssh: DEBUG: Command c127b556 on lago-he-basic-ansible-suite-master-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:d5c1c51b-f487-4b59-8079-bdb59bdfcfa7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:d5c1c51b-f487-4b59-8079-bdb59bdfcfa7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running c360ebd0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command c360ebd0 on lago-he-basic-ansible-suite-master-host-0 returned with 1
lago.ssh: DEBUG: Command c360ebd0 on lago-he-basic-ansible-suite-master-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:2dbd6214-28f6-4736-9a97-0e98076457c1:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:2dbd6214-28f6-4736-9a97-0e98076457c1:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running c58e6b3a on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command c58e6b3a on lago-he-basic-ansible-suite-master-host-0 returned with 1
lago.ssh: DEBUG: Command c58e6b3a on lago-he-basic-ansible-suite-master-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:502b040a-af3b-4f44-aed9-9dbce56933d2:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:502b040a-af3b-4f44-aed9-9dbce56933d2:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running c7b6c876 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command c7b6c876 on lago-he-basic-ansible-suite-master-host-0 returned with 1
lago.ssh: DEBUG: Command c7b6c876 on lago-he-basic-ansible-suite-master-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:51e6fde8-4042-4144-930e-252299ee405c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:51e6fde8-4042-4144-930e-252299ee405c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running c9e638c0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command c9e638c0 on lago-he-basic-ansible-suite-master-host-0 returned with 1
lago.ssh: DEBUG: Command c9e638c0 on lago-he-basic-ansible-suite-master-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:508cff8b-8c2a-4584-b493-d2679f10ad82:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:508cff8b-8c2a-4584-b493-d2679f10ad82:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running cc0c3816 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command cc0c3816 on lago-he-basic-ansible-suite-master-host-0 returned with 1
lago.ssh: DEBUG: Command cc0c3816 on lago-he-basic-ansible-suite-master-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:6033b3fb-f21b-447d-a137-04e7784aab4a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:6033b3fb-f21b-447d-a137-04e7784aab4a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running ce4de886 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command ce4de886 on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command ce4de886 on lago-he-basic-ansible-suite-master-host-0 output:
!! Cluster is in GLOBAL MAINTENANCE mode !!
--== Host lago-he-basic-ansible-suite-master-host-0 (id: 1) status ==--
conf_on_shared_storage : True
Status up-to-date : True
Hostname : lago-he-basic-ansible-suite-master-host-0
Host ID : 1
Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}
Score : 0
stopped : False
Local maintenance : False
crc32 : 1bfd8aa6
local_conf_timestamp : 2481
Host timestamp : 2481
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=2481 (Sat Sep 15 02:51:43 2018)
host-id=1
score=0
vm_conf_refresh_time=2481 (Sat Sep 15 02:51:43 2018)
conf_on_shared_storage=True
maintenance=False
state=ReinitializeFSM
stopped=False
--== Host lago-he-basic-ansible-suite-master-host-1 (id: 2) status ==--
conf_on_shared_storage : True
Status up-to-date : False
Hostname : lago-he-basic-ansible-suite-master-host-1
Host ID : 2
Engine status : unknown stale-data
Score : 3400
stopped : False
Local maintenance : False
crc32 : eb6c75a4
local_conf_timestamp : 2472
Host timestamp : 2472
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=2472 (Sat Sep 15 02:51:34 2018)
host-id=2
score=3400
vm_conf_refresh_time=2472 (Sat Sep 15 02:51:34 2018)
conf_on_shared_storage=True
maintenance=False
state=GlobalMaintenance
stopped=False
!! Cluster is in GLOBAL MAINTENANCE mode !!
root: INFO: * Agent is ready.
root: INFO: * Starting VM...
lago.ssh: DEBUG: start task:4b642e15-e2d0-441d-b6c7-6726e4671726:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:4b642e15-e2d0-441d-b6c7-6726e4671726:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running cecfa204 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-start
lago.ssh: DEBUG: Command cecfa204 on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command cecfa204 on lago-he-basic-ansible-suite-master-host-0 output:
VM exists and is down, cleaning up and restarting
root: INFO: * Command succeeded
root: INFO: * Waiting for VM to be UP...
lago.ssh: DEBUG: start task:46acaf72-3d13-4871-abd2-af9b3e9be8e0:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:46acaf72-3d13-4871-abd2-af9b3e9be8e0:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running d022e580 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command d022e580 on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command d022e580 on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2481 (Sat Sep 15 02:51:43 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=2481 (Sat Sep 15 02:51:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=ReinitializeFSM\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "1bfd8aa6", "local_conf_timestamp": 2481, "host-ts": 2481}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2472 (Sat Sep 15 02:51:34 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2472 (Sat Sep 15 02:51:34 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eb6c75a4", "local_conf_timestamp": 2472, "host-ts": 2472}, "global_maintenance": true}
lago.ssh: DEBUG: start task:9aacbec2-ebe2-4adb-980c-28909ccec04a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:9aacbec2-ebe2-4adb-980c-28909ccec04a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running d698d1ea on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command d698d1ea on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command d698d1ea on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2491 (Sat Sep 15 02:51:53 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2491 (Sat Sep 15 02:51:53 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e4fd4d62", "local_conf_timestamp": 2491, "host-ts": 2491}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2482 (Sat Sep 15 02:51:44 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2482 (Sat Sep 15 02:51:45 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "937b423f", "local_conf_timestamp": 2482, "host-ts": 2482}, "global_maintenance": true}
root: INFO: * VM is UP.
root: INFO: * Waiting for engine to start...
lago.ssh: DEBUG: start task:a1f19e92-cf1d-461b-8dfe-54f39840068c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:a1f19e92-cf1d-461b-8dfe-54f39840068c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running d7183cd2 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command d7183cd2 on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command d7183cd2 on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2491 (Sat Sep 15 02:51:53 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2491 (Sat Sep 15 02:51:53 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e4fd4d62", "local_conf_timestamp": 2491, "host-ts": 2491}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2482 (Sat Sep 15 02:51:44 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2482 (Sat Sep 15 02:51:45 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "937b423f", "local_conf_timestamp": 2482, "host-ts": 2482}, "global_maintenance": true}
lago.ssh: DEBUG: start task:7b048778-3a8f-468f-a3ba-40adba395ce2:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:7b048778-3a8f-468f-a3ba-40adba395ce2:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running dd98827e on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command dd98827e on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command dd98827e on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2501 (Sat Sep 15 02:52:03 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2502 (Sat Sep 15 02:52:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd877c52", "local_conf_timestamp": 2502, "host-ts": 2501}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2492 (Sat Sep 15 02:51:55 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2493 (Sat Sep 15 02:51:55 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ec7fddd4", "local_conf_timestamp": 2493, "host-ts": 2492}, "global_maintenance": true}
lago.ssh: DEBUG: start task:dec50117-7911-436c-92ab-23a6e01b9797:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:dec50117-7911-436c-92ab-23a6e01b9797:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running dfee1836 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command dfee1836 on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command dfee1836 on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2511 (Sat Sep 15 02:52:13 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2511 (Sat Sep 15 02:52:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c163b966", "local_conf_timestamp": 2511, "host-ts": 2511}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2492 (Sat Sep 15 02:51:55 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2493 (Sat Sep 15 02:51:55 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ec7fddd4", "local_conf_timestamp": 2493, "host-ts": 2492}, "global_maintenance": true}
lago.ssh: DEBUG: start task:3d54d54a-3318-45ce-9635-fa7c36c532e9:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:3d54d54a-3318-45ce-9635-fa7c36c532e9:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running e24c7e9c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command e24c7e9c on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command e24c7e9c on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2511 (Sat Sep 15 02:52:13 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2511 (Sat Sep 15 02:52:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c163b966", "local_conf_timestamp": 2511, "host-ts": 2511}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2502 (Sat Sep 15 02:52:04 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2502 (Sat Sep 15 02:52:04 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fdda37f9", "local_conf_timestamp": 2502, "host-ts": 2502}, "global_maintenance": true}
lago.ssh: DEBUG: start task:081e05c7-1491-4111-a828-dcdfb7750adb:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:081e05c7-1491-4111-a828-dcdfb7750adb:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running e4a9c406 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command e4a9c406 on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command e4a9c406 on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2511 (Sat Sep 15 02:52:13 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2511 (Sat Sep 15 02:52:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c163b966", "local_conf_timestamp": 2511, "host-ts": 2511}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2502 (Sat Sep 15 02:52:04 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2502 (Sat Sep 15 02:52:04 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fdda37f9", "local_conf_timestamp": 2502, "host-ts": 2502}, "global_maintenance": true}
lago.ssh: DEBUG: start task:ed1622e6-c317-4d65-878b-c034ae4c684a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:ed1622e6-c317-4d65-878b-c034ae4c684a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running e72300d0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command e72300d0 on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command e72300d0 on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2521 (Sat Sep 15 02:52:23 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2521 (Sat Sep 15 02:52:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "77dc31e3", "local_conf_timestamp": 2521, "host-ts": 2521}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2512 (Sat Sep 15 02:52:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2512 (Sat Sep 15 02:52:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f5d99884", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true}
lago.ssh: DEBUG: start task:b0e31724-b7d5-4e10-ba19-b85cd6a18237:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:b0e31724-b7d5-4e10-ba19-b85cd6a18237:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running e9946412 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command e9946412 on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command e9946412 on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2521 (Sat Sep 15 02:52:23 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2521 (Sat Sep 15 02:52:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "77dc31e3", "local_conf_timestamp": 2521, "host-ts": 2521}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2512 (Sat Sep 15 02:52:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2512 (Sat Sep 15 02:52:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f5d99884", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true}
lago.ssh: DEBUG: start task:a3e7508c-a4ab-4711-94c6-9274d4dd866a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:a3e7508c-a4ab-4711-94c6-9274d4dd866a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running ebdc77f0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command ebdc77f0 on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command ebdc77f0 on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2521 (Sat Sep 15 02:52:23 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2521 (Sat Sep 15 02:52:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "77dc31e3", "local_conf_timestamp": 2521, "host-ts": 2521}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2512 (Sat Sep 15 02:52:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2512 (Sat Sep 15 02:52:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f5d99884", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true}
lago.ssh: DEBUG: start task:07a8ca1d-6dae-49f5-9a41-8ff5113c3330:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:07a8ca1d-6dae-49f5-9a41-8ff5113c3330:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running ee2ea7e4 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command ee2ea7e4 on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command ee2ea7e4 on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2531 (Sat Sep 15 02:52:33 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2531 (Sat Sep 15 02:52:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "65a02926", "local_conf_timestamp": 2531, "host-ts": 2531}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2522 (Sat Sep 15 02:52:24 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2522 (Sat Sep 15 02:52:24 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eddd6903", "local_conf_timestamp": 2522, "host-ts": 2522}, "global_maintenance": true}
lago.ssh: DEBUG: start task:ff262ca1-ab96-4d86-b032-6684126b2551:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:ff262ca1-ab96-4d86-b032-6684126b2551:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running f06d4998 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command f06d4998 on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command f06d4998 on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2531 (Sat Sep 15 02:52:33 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2531 (Sat Sep 15 02:52:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "65a02926", "local_conf_timestamp": 2531, "host-ts": 2531}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2522 (Sat Sep 15 02:52:24 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2522 (Sat Sep 15 02:52:24 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eddd6903", "local_conf_timestamp": 2522, "host-ts": 2522}, "global_maintenance": true}
lago.ssh: DEBUG: start task:0bd6fcfe-a362-4cbb-9030-45e36f7121a3:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:0bd6fcfe-a362-4cbb-9030-45e36f7121a3:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running f2d07494 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command f2d07494 on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command f2d07494 on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2541 (Sat Sep 15 02:52:43 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2541 (Sat Sep 15 02:52:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ad4637d", "local_conf_timestamp": 2541, "host-ts": 2541}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2532 (Sat Sep 15 02:52:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2533 (Sat Sep 15 02:52:35 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "92d9f6e8", "local_conf_timestamp": 2533, "host-ts": 2532}, "global_maintenance": true}
lago.ssh: DEBUG: start task:7221ff8d-1909-4fcf-87c8-ef62ef65ed2c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:7221ff8d-1909-4fcf-87c8-ef62ef65ed2c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running f544a65a on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command f544a65a on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command f544a65a on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2541 (Sat Sep 15 02:52:43 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2541 (Sat Sep 15 02:52:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ad4637d", "local_conf_timestamp": 2541, "host-ts": 2541}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2532 (Sat Sep 15 02:52:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2533 (Sat Sep 15 02:52:35 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "92d9f6e8", "local_conf_timestamp": 2533, "host-ts": 2532}, "global_maintenance": true}
lago.ssh: DEBUG: start task:28d5ce55-c074-4848-aef7-6a9543351375:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:28d5ce55-c074-4848-aef7-6a9543351375:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running f7b68962 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command f7b68962 on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command f7b68962 on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2541 (Sat Sep 15 02:52:43 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2541 (Sat Sep 15 02:52:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ad4637d", "local_conf_timestamp": 2541, "host-ts": 2541}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2532 (Sat Sep 15 02:52:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2533 (Sat Sep 15 02:52:35 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "92d9f6e8", "local_conf_timestamp": 2533, "host-ts": 2532}, "global_maintenance": true}
lago.ssh: DEBUG: start task:70cda777-6455-4ebe-ad9e-c50880c6f5ce:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:70cda777-6455-4ebe-ad9e-c50880c6f5ce:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running fa0f7b06 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command fa0f7b06 on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command fa0f7b06 on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2551 (Sat Sep 15 02:52:53 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2551 (Sat Sep 15 02:52:53 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1890ef91", "local_conf_timestamp": 2551, "host-ts": 2551}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2542 (Sat Sep 15 02:52:44 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2542 (Sat Sep 15 02:52:44 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ddd48a0d", "local_conf_timestamp": 2542, "host-ts": 2542}, "global_maintenance": true}
lago.ssh: DEBUG: start task:c974937c-559f-41b0-b28b-dd0315b74d79:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:c974937c-559f-41b0-b28b-dd0315b74d79:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running fc7520bc on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command fc7520bc on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command fc7520bc on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2551 (Sat Sep 15 02:52:53 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2551 (Sat Sep 15 02:52:53 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1890ef91", "local_conf_timestamp": 2551, "host-ts": 2551}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2542 (Sat Sep 15 02:52:44 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2542 (Sat Sep 15 02:52:44 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ddd48a0d", "local_conf_timestamp": 2542, "host-ts": 2542}, "global_maintenance": true}
lago.ssh: DEBUG: start task:ccf674f8-d12b-4176-a81e-8fa962e61ea7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: end task:ccf674f8-d12b-4176-a81e-8fa962e61ea7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:
lago.ssh: DEBUG: Running fed087e8 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command fed087e8 on lago-he-basic-ansible-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command fed087e8 on lago-he-basic-ansible-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2561 (Sat Sep 15 02:53:03 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2561 (Sat Sep 15 02:53:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2e14c6de", "local_conf_timestamp": 2561, "host-ts": 2561}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2552 (Sat Sep 15 02:52:54 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2552 (Sat Sep 15 02:52:54 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-
ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7fbd8405db18>
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within
res = func()
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda>
for k, v in _get_he_status(host).items()
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status
raise RuntimeError('could not parse JSON: %s' % ret.out)
RuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2561 (Sat Sep 15 02:53:03 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2561 (Sat Sep 15 02:53:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2e14c6de", "local_conf_timestamp": 2561, "host-ts": 2561}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2552 (Sat Sep 15 02:52:54 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2552 (Sat Sep 15 02:52:54 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-
--------------------- >> end captured logging << ---------------------
Stack Trace:
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 53, in restart_he_vm
_wait_for_engine_health(host)
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 185, in _wait_for_engine_health
testlib.assert_true_within_long(lambda: any(
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long
assert_equals_within_long(func, True, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long
func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within
res = func()
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda>
for k, v in _get_he_status(host).items()
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status
raise RuntimeError('could not parse JSON: %s' % ret.out)
'could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2561 (Sat Sep 15 02:53:03 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2561 (Sat Sep 15 02:53:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2e14c6de", "local_conf_timestamp": 2561, "host-ts": 2561}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2552 (Sat Sep 15 02:52:54 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2552 (Sat Sep 15 02:52:54 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:ed47f4dc-58b7-4cd6-8718-820f742fbaa8:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:ed47f4dc-58b7-4cd6-8718-820f742fbaa8:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 9face572 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9face572 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 9face572 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2397 (Sat Sep 15 02:50:19 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2397 (Sat Sep 15 02:50:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5f097b86", "local_conf_timestamp": 2397, "host-ts": 2397}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2392 (Sat Sep 15 02:50:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2392 (Sat Sep 15 02:50:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "524cc788", "local_conf_timestamp": 2392, "host-ts": 2392}, "global_maintenance": true}\n\nroot: INFO: * Shutting down HE VM on host: lago-he-basic-ansible-suite-master-host-0\nlago.ssh: DEBUG: start task:9d2bbc93-b674-4f8f-9cff-0ed017480579:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:9d2bbc93-b674-4f8f-9cff-0ed017480579:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running a03b5ad2 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-shutdown\nlago.ssh: DEBUG: Command a03b5ad2 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be down...\nlago.ssh: DEBUG: start task:c6f14484-bc15-46ae-a11a-d71e7d7736e6:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:c6f14484-bc15-46ae-a11a-d71e7d7736e6:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running a0d7f720 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a0d7f720 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a0d7f720 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2397 (Sat Sep 15 02:50:19 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2397 (Sat Sep 15 02:50:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5f097b86", "local_conf_timestamp": 2397, "host-ts": 2397}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2392 (Sat Sep 15 02:50:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2392 (Sat Sep 15 02:50:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "524cc788", "local_conf_timestamp": 2392, "host-ts": 2392}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:b7f4b056-1ffd-4395-86d7-cbdba57b299a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:b7f4b056-1ffd-4395-86d7-cbdba57b299a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running a76cfd56 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a76cfd56 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a76cfd56 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2407 (Sat Sep 15 02:50:29 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2407 (Sat Sep 15 02:50:29 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f828e9c5", "local_conf_timestamp": 2407, "host-ts": 2407}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2402 (Sat Sep 15 02:50:24 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2402 (Sat Sep 15 02:50:25 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d36639d7", "local_conf_timestamp": 2402, "host-ts": 2402}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:0be59cd9-e4cb-4839-9891-894d2a5f611e:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:0be59cd9-e4cb-4839-9891-894d2a5f611e:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running a9bc073c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a9bc073c on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a9bc073c on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2417 (Sat Sep 15 02:50:39 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2417 (Sat Sep 15 02:50:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "42b65cd2", "local_conf_timestamp": 2417, "host-ts": 2417}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2412 (Sat Sep 15 02:50:34 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2412 (Sat Sep 15 02:50:34 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "db6596aa", "local_conf_timestamp": 2412, "host-ts": 2412}, "global_maintenance": true}\n\nroot: INFO: * VM is down.\nroot: INFO: * Stopping services...\nlago.ssh: DEBUG: start task:a7f11961-66fe-4a3e-8b5c-03640709deb3:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:a7f11961-66fe-4a3e-8b5c-03640709deb3:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running aa528b4e on lago-he-basic-ansible-suite-master-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command aa528b4e on lago-he-basic-ansible-suite-master-host-0 returned with 0\nroot: INFO: * Starting services...\nlago.ssh: DEBUG: start task:3470e088-3896-444c-9480-28f84251a111:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:3470e088-3896-444c-9480-28f84251a111:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running b42c375a on lago-he-basic-ansible-suite-master-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command b42c375a on lago-he-basic-ansible-suite-master-host-0 returned with 0\nroot: INFO: * Waiting for agent to be ready...\nlago.ssh: DEBUG: start task:c551d6c2-0baf-48c5-b907-f31f4391d1c6:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:c551d6c2-0baf-48c5-b907-f31f4391d1c6:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running b677b14c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command b677b14c on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command b677b14c on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:5dfa64c5-57dd-43f8-b09b-5e6549669059:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:5dfa64c5-57dd-43f8-b09b-5e6549669059:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running bcc603e6 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command bcc603e6 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command bcc603e6 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:4100a51d-ba55-4808-ac40-f68b20c91e05:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:4100a51d-ba55-4808-ac40-f68b20c91e05:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running beff4df2 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command beff4df2 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command beff4df2 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:46fa8f10-bbc9-433e-9b53-0029800375c7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:46fa8f10-bbc9-433e-9b53-0029800375c7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running c127b556 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command c127b556 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command c127b556 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:d5c1c51b-f487-4b59-8079-bdb59bdfcfa7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:d5c1c51b-f487-4b59-8079-bdb59bdfcfa7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running c360ebd0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command c360ebd0 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command c360ebd0 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:2dbd6214-28f6-4736-9a97-0e98076457c1:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:2dbd6214-28f6-4736-9a97-0e98076457c1:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running c58e6b3a on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command c58e6b3a on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command c58e6b3a on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:502b040a-af3b-4f44-aed9-9dbce56933d2:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:502b040a-af3b-4f44-aed9-9dbce56933d2:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running c7b6c876 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command c7b6c876 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command c7b6c876 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:51e6fde8-4042-4144-930e-252299ee405c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:51e6fde8-4042-4144-930e-252299ee405c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running c9e638c0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command c9e638c0 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command c9e638c0 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:508cff8b-8c2a-4584-b493-d2679f10ad82:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:508cff8b-8c2a-4584-b493-d2679f10ad82:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running cc0c3816 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command cc0c3816 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command cc0c3816 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:6033b3fb-f21b-447d-a137-04e7784aab4a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:6033b3fb-f21b-447d-a137-04e7784aab4a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running ce4de886 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command ce4de886 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ce4de886 on lago-he-basic-ansible-suite-master-host-0 output:\n \n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\n\n--== Host lago-he-basic-ansible-suite-master-host-0 (id: 1) status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : True\nHostname : lago-he-basic-ansible-suite-master-host-0\nHost ID : 1\nEngine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}\nScore : 0\nstopped : False\nLocal maintenance : False\ncrc32 : 1bfd8aa6\nlocal_conf_timestamp : 2481\nHost timestamp : 2481\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2481 (Sat Sep 15 02:51:43 2018)\n\thost-id=1\n\tscore=0\n\tvm_conf_refresh_time=2481 (Sat Sep 15 02:51:43 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=ReinitializeFSM\n\tstopped=False\n\n\n--== Host lago-he-basic-ansible-suite-master-host-1 (id: 2) status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : False\nHostname : lago-he-basic-ansible-suite-master-host-1\nHost ID : 2\nEngine status : unknown stale-data\nScore : 3400\nstopped : False\nLocal maintenance : False\ncrc32 : eb6c75a4\nlocal_conf_timestamp : 2472\nHost timestamp : 2472\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2472 (Sat Sep 15 02:51:34 2018)\n\thost-id=2\n\tscore=3400\n\tvm_conf_refresh_time=2472 (Sat Sep 15 02:51:34 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=GlobalMaintenance\n\tstopped=False\n\n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\nroot: INFO: * Agent is ready.\nroot: INFO: * Starting VM...\nlago.ssh: DEBUG: start task:4b642e15-e2d0-441d-b6c7-6726e4671726:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:4b642e15-e2d0-441d-b6c7-6726e4671726:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running cecfa204 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-start\nlago.ssh: DEBUG: Command cecfa204 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command cecfa204 on lago-he-basic-ansible-suite-master-host-0 output:\n VM exists and is down, cleaning up and restarting\n\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be UP...\nlago.ssh: DEBUG: start task:46acaf72-3d13-4871-abd2-af9b3e9be8e0:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:46acaf72-3d13-4871-abd2-af9b3e9be8e0:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running d022e580 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command d022e580 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command d022e580 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2481 (Sat Sep 15 02:51:43 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=2481 (Sat Sep 15 02:51:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=ReinitializeFSM\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "1bfd8aa6", "local_conf_timestamp": 2481, "host-ts": 2481}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2472 (Sat Sep 15 02:51:34 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2472 (Sat Sep 15 02:51:34 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eb6c75a4", "local_conf_timestamp": 2472, "host-ts": 2472}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:9aacbec2-ebe2-4adb-980c-28909ccec04a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:9aacbec2-ebe2-4adb-980c-28909ccec04a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running d698d1ea on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command d698d1ea on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command d698d1ea on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2491 (Sat Sep 15 02:51:53 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2491 (Sat Sep 15 02:51:53 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e4fd4d62", "local_conf_timestamp": 2491, "host-ts": 2491}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2482 (Sat Sep 15 02:51:44 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2482 (Sat Sep 15 02:51:45 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "937b423f", "local_conf_timestamp": 2482, "host-ts": 2482}, "global_maintenance": true}\n\nroot: INFO: * VM is UP.\nroot: INFO: * Waiting for engine to start...\nlago.ssh: DEBUG: start task:a1f19e92-cf1d-461b-8dfe-54f39840068c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:a1f19e92-cf1d-461b-8dfe-54f39840068c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running d7183cd2 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command d7183cd2 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command d7183cd2 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2491 (Sat Sep 15 02:51:53 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2491 (Sat Sep 15 02:51:53 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e4fd4d62", "local_conf_timestamp": 2491, "host-ts": 2491}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2482 (Sat Sep 15 02:51:44 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2482 (Sat Sep 15 02:51:45 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "937b423f", "local_conf_timestamp": 2482, "host-ts": 2482}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:7b048778-3a8f-468f-a3ba-40adba395ce2:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:7b048778-3a8f-468f-a3ba-40adba395ce2:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running dd98827e on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command dd98827e on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command dd98827e on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2501 (Sat Sep 15 02:52:03 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2502 (Sat Sep 15 02:52:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd877c52", "local_conf_timestamp": 2502, "host-ts": 2501}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2492 (Sat Sep 15 02:51:55 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2493 (Sat Sep 15 02:51:55 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ec7fddd4", "local_conf_timestamp": 2493, "host-ts": 2492}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:dec50117-7911-436c-92ab-23a6e01b9797:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:dec50117-7911-436c-92ab-23a6e01b9797:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running dfee1836 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command dfee1836 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command dfee1836 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2511 (Sat Sep 15 02:52:13 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2511 (Sat Sep 15 02:52:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c163b966", "local_conf_timestamp": 2511, "host-ts": 2511}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2492 (Sat Sep 15 02:51:55 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2493 (Sat Sep 15 02:51:55 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ec7fddd4", "local_conf_timestamp": 2493, "host-ts": 2492}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:3d54d54a-3318-45ce-9635-fa7c36c532e9:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:3d54d54a-3318-45ce-9635-fa7c36c532e9:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running e24c7e9c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e24c7e9c on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command e24c7e9c on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2511 (Sat Sep 15 02:52:13 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2511 (Sat Sep 15 02:52:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c163b966", "local_conf_timestamp": 2511, "host-ts": 2511}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2502 (Sat Sep 15 02:52:04 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2502 (Sat Sep 15 02:52:04 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fdda37f9", "local_conf_timestamp": 2502, "host-ts": 2502}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:081e05c7-1491-4111-a828-dcdfb7750adb:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:081e05c7-1491-4111-a828-dcdfb7750adb:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running e4a9c406 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e4a9c406 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command e4a9c406 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2511 (Sat Sep 15 02:52:13 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2511 (Sat Sep 15 02:52:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c163b966", "local_conf_timestamp": 2511, "host-ts": 2511}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2502 (Sat Sep 15 02:52:04 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2502 (Sat Sep 15 02:52:04 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fdda37f9", "local_conf_timestamp": 2502, "host-ts": 2502}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ed1622e6-c317-4d65-878b-c034ae4c684a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:ed1622e6-c317-4d65-878b-c034ae4c684a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running e72300d0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e72300d0 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command e72300d0 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2521 (Sat Sep 15 02:52:23 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2521 (Sat Sep 15 02:52:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "77dc31e3", "local_conf_timestamp": 2521, "host-ts": 2521}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2512 (Sat Sep 15 02:52:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2512 (Sat Sep 15 02:52:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f5d99884", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:b0e31724-b7d5-4e10-ba19-b85cd6a18237:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:b0e31724-b7d5-4e10-ba19-b85cd6a18237:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running e9946412 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e9946412 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command e9946412 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2521 (Sat Sep 15 02:52:23 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2521 (Sat Sep 15 02:52:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "77dc31e3", "local_conf_timestamp": 2521, "host-ts": 2521}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2512 (Sat Sep 15 02:52:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2512 (Sat Sep 15 02:52:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f5d99884", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:a3e7508c-a4ab-4711-94c6-9274d4dd866a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:a3e7508c-a4ab-4711-94c6-9274d4dd866a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running ebdc77f0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ebdc77f0 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ebdc77f0 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2521 (Sat Sep 15 02:52:23 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2521 (Sat Sep 15 02:52:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "77dc31e3", "local_conf_timestamp": 2521, "host-ts": 2521}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2512 (Sat Sep 15 02:52:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2512 (Sat Sep 15 02:52:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f5d99884", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:07a8ca1d-6dae-49f5-9a41-8ff5113c3330:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:07a8ca1d-6dae-49f5-9a41-8ff5113c3330:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running ee2ea7e4 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ee2ea7e4 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ee2ea7e4 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2531 (Sat Sep 15 02:52:33 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2531 (Sat Sep 15 02:52:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "65a02926", "local_conf_timestamp": 2531, "host-ts": 2531}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2522 (Sat Sep 15 02:52:24 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2522 (Sat Sep 15 02:52:24 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eddd6903", "local_conf_timestamp": 2522, "host-ts": 2522}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ff262ca1-ab96-4d86-b032-6684126b2551:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:ff262ca1-ab96-4d86-b032-6684126b2551:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running f06d4998 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f06d4998 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command f06d4998 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2531 (Sat Sep 15 02:52:33 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2531 (Sat Sep 15 02:52:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "65a02926", "local_conf_timestamp": 2531, "host-ts": 2531}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2522 (Sat Sep 15 02:52:24 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2522 (Sat Sep 15 02:52:24 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eddd6903", "local_conf_timestamp": 2522, "host-ts": 2522}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:0bd6fcfe-a362-4cbb-9030-45e36f7121a3:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:0bd6fcfe-a362-4cbb-9030-45e36f7121a3:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running f2d07494 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f2d07494 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command f2d07494 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2541 (Sat Sep 15 02:52:43 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2541 (Sat Sep 15 02:52:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ad4637d", "local_conf_timestamp": 2541, "host-ts": 2541}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2532 (Sat Sep 15 02:52:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2533 (Sat Sep 15 02:52:35 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "92d9f6e8", "local_conf_timestamp": 2533, "host-ts": 2532}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:7221ff8d-1909-4fcf-87c8-ef62ef65ed2c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:7221ff8d-1909-4fcf-87c8-ef62ef65ed2c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running f544a65a on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f544a65a on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command f544a65a on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2541 (Sat Sep 15 02:52:43 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2541 (Sat Sep 15 02:52:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ad4637d", "local_conf_timestamp": 2541, "host-ts": 2541}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2532 (Sat Sep 15 02:52:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2533 (Sat Sep 15 02:52:35 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "92d9f6e8", "local_conf_timestamp": 2533, "host-ts": 2532}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:28d5ce55-c074-4848-aef7-6a9543351375:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:28d5ce55-c074-4848-aef7-6a9543351375:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running f7b68962 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f7b68962 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command f7b68962 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2541 (Sat Sep 15 02:52:43 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2541 (Sat Sep 15 02:52:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ad4637d", "local_conf_timestamp": 2541, "host-ts": 2541}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2532 (Sat Sep 15 02:52:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2533 (Sat Sep 15 02:52:35 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "92d9f6e8", "local_conf_timestamp": 2533, "host-ts": 2532}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:70cda777-6455-4ebe-ad9e-c50880c6f5ce:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:70cda777-6455-4ebe-ad9e-c50880c6f5ce:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running fa0f7b06 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fa0f7b06 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command fa0f7b06 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2551 (Sat Sep 15 02:52:53 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2551 (Sat Sep 15 02:52:53 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1890ef91", "local_conf_timestamp": 2551, "host-ts": 2551}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2542 (Sat Sep 15 02:52:44 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2542 (Sat Sep 15 02:52:44 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ddd48a0d", "local_conf_timestamp": 2542, "host-ts": 2542}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:c974937c-559f-41b0-b28b-dd0315b74d79:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:c974937c-559f-41b0-b28b-dd0315b74d79:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running fc7520bc on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fc7520bc on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command fc7520bc on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2551 (Sat Sep 15 02:52:53 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2551 (Sat Sep 15 02:52:53 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1890ef91", "local_conf_timestamp": 2551, "host-ts": 2551}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2542 (Sat Sep 15 02:52:44 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2542 (Sat Sep 15 02:52:44 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ddd48a0d", "local_conf_timestamp": 2542, "host-ts": 2542}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ccf674f8-d12b-4176-a81e-8fa962e61ea7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:ccf674f8-d12b-4176-a81e-8fa962e61ea7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running fed087e8 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fed087e8 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command fed087e8 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2561 (Sat Sep 15 02:53:03 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2561 (Sat Sep 15 02:53:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2e14c6de", "local_conf_timestamp": 2561, "host-ts": 2561}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2552 (Sat Sep 15 02:52:54 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2552 (Sat Sep 15 02:52:54 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7fbd8405db18>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda>\n for k, v in _get_he_status(host).items()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status\n raise RuntimeError(\'could not parse JSON: %s\' % ret.out)\nRuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2561 (Sat Sep 15 02:53:03 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2561 (Sat Sep 15 02:53:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2e14c6de", "local_conf_timestamp": 2561, "host-ts": 2561}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2552 (Sat Sep 15 02:52:54 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2552 (Sat Sep 15 02:52:54 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-\n--------------------- >> end captured logging << ---------------------'
6 years, 2 months
Build failed in Jenkins:
system-sync_mirrors-centos-updates-el7-x86_64 #1946
by jenkins@jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...>
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...>
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 9c187d6e9145db35423f495b553304ceb75489cb (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 9c187d6e9145db35423f495b553304ceb75489cb
Commit message: "Add jobs for ovs-cni"
> git rev-list --no-walk 9c187d6e9145db35423f495b553304ceb75489cb # timeout=10
[system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins4484725594797596479.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf
Checking if mirror needs a resync
Traceback (most recent call last):
File "/usr/bin/reposync", line 343, in <module>
main()
File "/usr/bin/reposync", line 175, in main
my.doRepoSetup()
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
return self._getRepos(thisrepo, True)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
self._repos.doSetup(thisrepo)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
dl = repo._async and repo._commonLoadRepoXML(repo)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML
local = self.cachedir + '/repomd.xml'
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda>
cachedir = property(lambda self: self._dirGetAttr('cachedir'))
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr
self.dirSetup()
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup
self._dirSetupMkdir_p(dir)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p
raise Errors.RepoError, msg
yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7/packages error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7/packages'
Build step 'Execute shell' marked build as failure
6 years, 2 months