September 2018 - Infra - oVirt List Archives

[JIRA] (OVIRT-2499) Add support for tag stage in STDCI
by Daniel Belenky (oVirt JIRA) 16 Sep '18

16 Sep '18

Daniel Belenky created OVIRT-2499: ------------------------------------- Summary: Add support for tag stage in STDCI Key: OVIRT-2499 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2499 Project: oVirt - virtualization made easy Issue Type: Epic Components: Standard CI (Pipelines), STDCI DSL Reporter: Daniel Belenky Assignee: infra As a developer, I want to have the ability to run my automation code upon new tags being pushed to my project so that I can trigger an automated release process. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[oVirt Jenkins] ovirt-system-tests_network-suite-4.2 - Build # 581 - Failure!
by jenkins＠jenkins.phx.ovirt.org 16 Sep '18

16 Sep '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/581/ Build Number: 581 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #581 [Milan Zamazal] Update test-vm.ovf file ----------------- Failed Tests: ----------------- No tests ran.

1 1

[CQ]: 94337, 2 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 16 Sep '18

16 Sep '18

A system test invoked by the "ovirt-master" change queue including change 94337,2 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 94081,3 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 94081,3 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/94337/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/94081/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10326/

1 0

[JIRA] (OVIRT-2498) Failing KubeVirt CI
by Barak Korren (oVirt JIRA) 16 Sep '18

16 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2498?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-2498: ------------------------------------- [~pkotas] I think the best place to discuss Kubevirt issues is on the Kubevirt-related mailing lists where other Kubvirt developers can see the discussion. To your questions: {quote} I am working on fixing the issues on the KubeVirt e2e test suites. This task is directly related to unstable CI, due to unknown errors. The progress is reported in the CNV trello: https://trello.com/c/HNXcMEQu/161-epic-improve-ci I am creating this issue since the KubeVirt experience random timeouts on random tests most of the times when test suites run. The issue from outside is showing as timeouts on difference part of tests. Sometimes the tests fails in set up phase, again due to random timeout. The example in the link bellow timed out for network connection on localhost. [check-patch.k8s-1.11.0-dev.el7.x86_64] requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60) {quote} Its generally a bad idea to rely to much on timeout in a test suit like this. We've seen such issues over and over again in OST as well. Its probably best to remove all such timeout definitions and just have overall timeout set for the entire test suit. {quote} Example of failing test suites is here https://jenkins.ovirt.org/job/kubevirt_kubevirt_standard-check-pr/1916/cons… The list of errors related to the failing CI can be found in my notes https://docs.google.com/document/d/1_ll1DOMHgCRHn_Df9i4uvtRFyMK-bDCHEeGfJFT… I am not sure whether KubeVirt already shared the resource requirements, so I provide short summary: Resources for KubeVirt e2e tests: at least 12GB of RAM - we start 3 nodes (3 docker images) each require 4GB of RAM exposed /dev/kvm to enable native virtualization cached images, since these are used to build the test cluster: kubevirtci/os-3.10.0-crio:latest kubevirtci/os-3.10.0-multus:latest kubevirtci/os-3.10.0:latest kubevirtci/k8s-1.10.4:latest kubevirtci/k8s-multus-1.11.1:latest kubevirtci/k8s-1.11.0:latest How can we overcome this? Can we work together to build a suitable requirements for running the tests so it passes each time? {quote} To my knowledge the existing setup meets all the requirements you specify above. We have 3 physical hosts that are used to run Kubevirt tests, each host has 128GB of ram and runs 7 containers where each container runs its own Libvirt, Docker and Systemd so that it looks like its own host to the tests running inside. The amount of containers per host was calculated to have each container have a little over 16GB of RAM for itself. So we should have more then enough for Kubevirt. Also, in our measurements Kubevirt's CI tests took way less then 12GB, ansd were somewhere around 8GB. All the images that start with 'kubevirtci' are cached by the system. WRT /dev/kvm - we do have it exposed in the containers we run, but I think that is irrelevant since AFAIK Kubevirt-CI runs qemu on its own inside its own container, so the /dev/kvm device files simply needs to exist inside that container. > Failing KubeVirt CI > ------------------- > > Key: OVIRT-2498 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2498 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Petr Kotas > Assignee: infra > > Hi, > I am working on fixing the issues on the KubeVirt e2e test suites. This > task is directly related to unstable CI, due to unknown errors. > The progress is reported in the CNV trello: > https://trello.com/c/HNXcMEQu/161-epic-improve-ci > I am creating this issue since the KubeVirt experience random timeouts on > random tests most of the times when test suites run. > The issue from outside is showing as timeouts on difference part of tests. > Sometimes the tests fails in set up phase, again due to random timeout. > The example in the link bellow timed out for network connection on > localhost. > [check-patch.k8s-1.11.0-dev.el7.x86_64] > requests.exceptions.ReadTimeout: > UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. > (read timeout=60) > Example of failing test suites is here > https://jenkins.ovirt.org/job/kubevirt_kubevirt_standard-check-pr/1916/cons… > The list of errors related to the failing CI can be found in my notes > https://docs.google.com/document/d/1_ll1DOMHgCRHn_Df9i4uvtRFyMK-bDCHEeGfJFT… > I am not sure whether KubeVirt already shared the resource requirements, so > I provide short summary: > *Resources for KubeVirt e2e tests:* > - at least 12GB of RAM - we start 3 nodes (3 docker images) each require > 4GB of RAM > - exposed /dev/kvm to enable native virtualization > - cached images, since these are used to build the test cluster: > - kubevirtci/os-3.10.0-crio:latest > - kubevirtci/os-3.10.0-multus:latest > - kubevirtci/os-3.10.0:latest > - kubevirtci/k8s-1.10.4:latest > - kubevirtci/k8s-multus-1.11.1:latest > - kubevirtci/k8s-1.11.0:latest > How can we overcome this? Can we work together to build a suitable > requirements for running the tests so it passes each time? > Kind regards, > Petr Kotas -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

oVirt infra daily report - unstable production jobs - 584
by jenkins＠jenkins.phx.ovirt.org 16 Sep '18

16 Sep '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/584//artifact/exported-a… Cheers, Jenkins

1 0

[oVirt Jenkins] ovirt-system-tests_he-basic-ansible-suite-master - Build # 632 - Failure!
by jenkins＠jenkins.phx.ovirt.org 15 Sep '18

15 Sep '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-mast… Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-mast… Build Number: 632 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #632 [Milan Zamazal] Update test-vm.ovf file ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 008_restart_he_vm.restart_he_vm Error Message: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2561 (Sat Sep 15 02:53:03 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2561 (Sat Sep 15 02:53:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2e14c6de", "local_conf_timestamp": 2561, "host-ts": 2561}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2552 (Sat Sep 15 02:52:54 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2552 (Sat Sep 15 02:52:54 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host- -------------------- >> begin captured logging << -------------------- lago.ssh: DEBUG: start task:ed47f4dc-58b7-4cd6-8718-820f742fbaa8:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:ed47f4dc-58b7-4cd6-8718-820f742fbaa8:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 9face572 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9face572 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 9face572 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2397 (Sat Sep 15 02:50:19 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2397 (Sat Sep 15 02:50:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5f097b86", "local_conf_timestamp": 2397, "host-ts": 2397}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2392 (Sat Sep 15 02:50:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2392 (Sat Sep 15 02:50:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "524cc788", "local_conf_timestamp": 2392, "host-ts": 2392}, "global_maintenance": true} root: INFO: * Shutting down HE VM on host: lago-he-basic-ansible-suite-master-host-0 lago.ssh: DEBUG: start task:9d2bbc93-b674-4f8f-9cff-0ed017480579:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:9d2bbc93-b674-4f8f-9cff-0ed017480579:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running a03b5ad2 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-shutdown lago.ssh: DEBUG: Command a03b5ad2 on lago-he-basic-ansible-suite-master-host-0 returned with 0 root: INFO: * Command succeeded root: INFO: * Waiting for VM to be down... lago.ssh: DEBUG: start task:c6f14484-bc15-46ae-a11a-d71e7d7736e6:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:c6f14484-bc15-46ae-a11a-d71e7d7736e6:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running a0d7f720 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a0d7f720 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command a0d7f720 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2397 (Sat Sep 15 02:50:19 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2397 (Sat Sep 15 02:50:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5f097b86", "local_conf_timestamp": 2397, "host-ts": 2397}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2392 (Sat Sep 15 02:50:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2392 (Sat Sep 15 02:50:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "524cc788", "local_conf_timestamp": 2392, "host-ts": 2392}, "global_maintenance": true} lago.ssh: DEBUG: start task:b7f4b056-1ffd-4395-86d7-cbdba57b299a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:b7f4b056-1ffd-4395-86d7-cbdba57b299a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running a76cfd56 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a76cfd56 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command a76cfd56 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2407 (Sat Sep 15 02:50:29 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2407 (Sat Sep 15 02:50:29 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f828e9c5", "local_conf_timestamp": 2407, "host-ts": 2407}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2402 (Sat Sep 15 02:50:24 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2402 (Sat Sep 15 02:50:25 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d36639d7", "local_conf_timestamp": 2402, "host-ts": 2402}, "global_maintenance": true} lago.ssh: DEBUG: start task:0be59cd9-e4cb-4839-9891-894d2a5f611e:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:0be59cd9-e4cb-4839-9891-894d2a5f611e:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running a9bc073c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a9bc073c on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command a9bc073c on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2417 (Sat Sep 15 02:50:39 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2417 (Sat Sep 15 02:50:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "42b65cd2", "local_conf_timestamp": 2417, "host-ts": 2417}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2412 (Sat Sep 15 02:50:34 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2412 (Sat Sep 15 02:50:34 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "db6596aa", "local_conf_timestamp": 2412, "host-ts": 2412}, "global_maintenance": true} root: INFO: * VM is down. root: INFO: * Stopping services... lago.ssh: DEBUG: start task:a7f11961-66fe-4a3e-8b5c-03640709deb3:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:a7f11961-66fe-4a3e-8b5c-03640709deb3:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running aa528b4e on lago-he-basic-ansible-suite-master-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command aa528b4e on lago-he-basic-ansible-suite-master-host-0 returned with 0 root: INFO: * Starting services... lago.ssh: DEBUG: start task:3470e088-3896-444c-9480-28f84251a111:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:3470e088-3896-444c-9480-28f84251a111:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running b42c375a on lago-he-basic-ansible-suite-master-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command b42c375a on lago-he-basic-ansible-suite-master-host-0 returned with 0 root: INFO: * Waiting for agent to be ready... lago.ssh: DEBUG: start task:c551d6c2-0baf-48c5-b907-f31f4391d1c6:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:c551d6c2-0baf-48c5-b907-f31f4391d1c6:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running b677b14c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command b677b14c on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command b677b14c on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:5dfa64c5-57dd-43f8-b09b-5e6549669059:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:5dfa64c5-57dd-43f8-b09b-5e6549669059:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running bcc603e6 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command bcc603e6 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command bcc603e6 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:4100a51d-ba55-4808-ac40-f68b20c91e05:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:4100a51d-ba55-4808-ac40-f68b20c91e05:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running beff4df2 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command beff4df2 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command beff4df2 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:46fa8f10-bbc9-433e-9b53-0029800375c7:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:46fa8f10-bbc9-433e-9b53-0029800375c7:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running c127b556 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command c127b556 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command c127b556 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:d5c1c51b-f487-4b59-8079-bdb59bdfcfa7:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:d5c1c51b-f487-4b59-8079-bdb59bdfcfa7:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running c360ebd0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command c360ebd0 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command c360ebd0 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:2dbd6214-28f6-4736-9a97-0e98076457c1:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:2dbd6214-28f6-4736-9a97-0e98076457c1:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running c58e6b3a on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command c58e6b3a on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command c58e6b3a on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:502b040a-af3b-4f44-aed9-9dbce56933d2:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:502b040a-af3b-4f44-aed9-9dbce56933d2:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running c7b6c876 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command c7b6c876 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command c7b6c876 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:51e6fde8-4042-4144-930e-252299ee405c:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:51e6fde8-4042-4144-930e-252299ee405c:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running c9e638c0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command c9e638c0 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command c9e638c0 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:508cff8b-8c2a-4584-b493-d2679f10ad82:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:508cff8b-8c2a-4584-b493-d2679f10ad82:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running cc0c3816 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command cc0c3816 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command cc0c3816 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:6033b3fb-f21b-447d-a137-04e7784aab4a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:6033b3fb-f21b-447d-a137-04e7784aab4a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running ce4de886 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command ce4de886 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command ce4de886 on lago-he-basic-ansible-suite-master-host-0 output: !! Cluster is in GLOBAL MAINTENANCE mode !! --== Host lago-he-basic-ansible-suite-master-host-0 (id: 1) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : lago-he-basic-ansible-suite-master-host-0 Host ID : 1 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"} Score : 0 stopped : False Local maintenance : False crc32 : 1bfd8aa6 local_conf_timestamp : 2481 Host timestamp : 2481 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2481 (Sat Sep 15 02:51:43 2018) host-id=1 score=0 vm_conf_refresh_time=2481 (Sat Sep 15 02:51:43 2018) conf_on_shared_storage=True maintenance=False state=ReinitializeFSM stopped=False --== Host lago-he-basic-ansible-suite-master-host-1 (id: 2) status ==-- conf_on_shared_storage : True Status up-to-date : False Hostname : lago-he-basic-ansible-suite-master-host-1 Host ID : 2 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : eb6c75a4 local_conf_timestamp : 2472 Host timestamp : 2472 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2472 (Sat Sep 15 02:51:34 2018) host-id=2 score=3400 vm_conf_refresh_time=2472 (Sat Sep 15 02:51:34 2018) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False !! Cluster is in GLOBAL MAINTENANCE mode !! root: INFO: * Agent is ready. root: INFO: * Starting VM... lago.ssh: DEBUG: start task:4b642e15-e2d0-441d-b6c7-6726e4671726:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:4b642e15-e2d0-441d-b6c7-6726e4671726:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running cecfa204 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-start lago.ssh: DEBUG: Command cecfa204 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command cecfa204 on lago-he-basic-ansible-suite-master-host-0 output: VM exists and is down, cleaning up and restarting root: INFO: * Command succeeded root: INFO: * Waiting for VM to be UP... lago.ssh: DEBUG: start task:46acaf72-3d13-4871-abd2-af9b3e9be8e0:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:46acaf72-3d13-4871-abd2-af9b3e9be8e0:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running d022e580 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command d022e580 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command d022e580 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2481 (Sat Sep 15 02:51:43 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=2481 (Sat Sep 15 02:51:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=ReinitializeFSM\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "1bfd8aa6", "local_conf_timestamp": 2481, "host-ts": 2481}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2472 (Sat Sep 15 02:51:34 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2472 (Sat Sep 15 02:51:34 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eb6c75a4", "local_conf_timestamp": 2472, "host-ts": 2472}, "global_maintenance": true} lago.ssh: DEBUG: start task:9aacbec2-ebe2-4adb-980c-28909ccec04a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:9aacbec2-ebe2-4adb-980c-28909ccec04a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running d698d1ea on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command d698d1ea on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command d698d1ea on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2491 (Sat Sep 15 02:51:53 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2491 (Sat Sep 15 02:51:53 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e4fd4d62", "local_conf_timestamp": 2491, "host-ts": 2491}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2482 (Sat Sep 15 02:51:44 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2482 (Sat Sep 15 02:51:45 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "937b423f", "local_conf_timestamp": 2482, "host-ts": 2482}, "global_maintenance": true} root: INFO: * VM is UP. root: INFO: * Waiting for engine to start... lago.ssh: DEBUG: start task:a1f19e92-cf1d-461b-8dfe-54f39840068c:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:a1f19e92-cf1d-461b-8dfe-54f39840068c:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running d7183cd2 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command d7183cd2 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command d7183cd2 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2491 (Sat Sep 15 02:51:53 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2491 (Sat Sep 15 02:51:53 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e4fd4d62", "local_conf_timestamp": 2491, "host-ts": 2491}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2482 (Sat Sep 15 02:51:44 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2482 (Sat Sep 15 02:51:45 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "937b423f", "local_conf_timestamp": 2482, "host-ts": 2482}, "global_maintenance": true} lago.ssh: DEBUG: start task:7b048778-3a8f-468f-a3ba-40adba395ce2:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:7b048778-3a8f-468f-a3ba-40adba395ce2:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running dd98827e on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command dd98827e on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command dd98827e on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2501 (Sat Sep 15 02:52:03 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2502 (Sat Sep 15 02:52:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd877c52", "local_conf_timestamp": 2502, "host-ts": 2501}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2492 (Sat Sep 15 02:51:55 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2493 (Sat Sep 15 02:51:55 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ec7fddd4", "local_conf_timestamp": 2493, "host-ts": 2492}, "global_maintenance": true} lago.ssh: DEBUG: start task:dec50117-7911-436c-92ab-23a6e01b9797:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:dec50117-7911-436c-92ab-23a6e01b9797:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running dfee1836 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command dfee1836 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command dfee1836 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2511 (Sat Sep 15 02:52:13 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2511 (Sat Sep 15 02:52:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c163b966", "local_conf_timestamp": 2511, "host-ts": 2511}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2492 (Sat Sep 15 02:51:55 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2493 (Sat Sep 15 02:51:55 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ec7fddd4", "local_conf_timestamp": 2493, "host-ts": 2492}, "global_maintenance": true} lago.ssh: DEBUG: start task:3d54d54a-3318-45ce-9635-fa7c36c532e9:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:3d54d54a-3318-45ce-9635-fa7c36c532e9:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running e24c7e9c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e24c7e9c on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command e24c7e9c on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2511 (Sat Sep 15 02:52:13 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2511 (Sat Sep 15 02:52:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c163b966", "local_conf_timestamp": 2511, "host-ts": 2511}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2502 (Sat Sep 15 02:52:04 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2502 (Sat Sep 15 02:52:04 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fdda37f9", "local_conf_timestamp": 2502, "host-ts": 2502}, "global_maintenance": true} lago.ssh: DEBUG: start task:081e05c7-1491-4111-a828-dcdfb7750adb:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:081e05c7-1491-4111-a828-dcdfb7750adb:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running e4a9c406 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e4a9c406 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command e4a9c406 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2511 (Sat Sep 15 02:52:13 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2511 (Sat Sep 15 02:52:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c163b966", "local_conf_timestamp": 2511, "host-ts": 2511}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2502 (Sat Sep 15 02:52:04 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2502 (Sat Sep 15 02:52:04 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fdda37f9", "local_conf_timestamp": 2502, "host-ts": 2502}, "global_maintenance": true} lago.ssh: DEBUG: start task:ed1622e6-c317-4d65-878b-c034ae4c684a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:ed1622e6-c317-4d65-878b-c034ae4c684a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running e72300d0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e72300d0 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command e72300d0 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2521 (Sat Sep 15 02:52:23 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2521 (Sat Sep 15 02:52:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "77dc31e3", "local_conf_timestamp": 2521, "host-ts": 2521}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2512 (Sat Sep 15 02:52:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2512 (Sat Sep 15 02:52:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f5d99884", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true} lago.ssh: DEBUG: start task:b0e31724-b7d5-4e10-ba19-b85cd6a18237:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:b0e31724-b7d5-4e10-ba19-b85cd6a18237:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running e9946412 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e9946412 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command e9946412 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2521 (Sat Sep 15 02:52:23 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2521 (Sat Sep 15 02:52:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "77dc31e3", "local_conf_timestamp": 2521, "host-ts": 2521}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2512 (Sat Sep 15 02:52:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2512 (Sat Sep 15 02:52:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f5d99884", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true} lago.ssh: DEBUG: start task:a3e7508c-a4ab-4711-94c6-9274d4dd866a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:a3e7508c-a4ab-4711-94c6-9274d4dd866a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running ebdc77f0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ebdc77f0 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command ebdc77f0 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2521 (Sat Sep 15 02:52:23 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2521 (Sat Sep 15 02:52:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "77dc31e3", "local_conf_timestamp": 2521, "host-ts": 2521}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2512 (Sat Sep 15 02:52:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2512 (Sat Sep 15 02:52:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f5d99884", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true} lago.ssh: DEBUG: start task:07a8ca1d-6dae-49f5-9a41-8ff5113c3330:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:07a8ca1d-6dae-49f5-9a41-8ff5113c3330:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running ee2ea7e4 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ee2ea7e4 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command ee2ea7e4 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2531 (Sat Sep 15 02:52:33 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2531 (Sat Sep 15 02:52:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "65a02926", "local_conf_timestamp": 2531, "host-ts": 2531}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2522 (Sat Sep 15 02:52:24 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2522 (Sat Sep 15 02:52:24 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eddd6903", "local_conf_timestamp": 2522, "host-ts": 2522}, "global_maintenance": true} lago.ssh: DEBUG: start task:ff262ca1-ab96-4d86-b032-6684126b2551:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:ff262ca1-ab96-4d86-b032-6684126b2551:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running f06d4998 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f06d4998 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command f06d4998 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2531 (Sat Sep 15 02:52:33 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2531 (Sat Sep 15 02:52:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "65a02926", "local_conf_timestamp": 2531, "host-ts": 2531}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2522 (Sat Sep 15 02:52:24 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2522 (Sat Sep 15 02:52:24 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eddd6903", "local_conf_timestamp": 2522, "host-ts": 2522}, "global_maintenance": true} lago.ssh: DEBUG: start task:0bd6fcfe-a362-4cbb-9030-45e36f7121a3:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:0bd6fcfe-a362-4cbb-9030-45e36f7121a3:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running f2d07494 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f2d07494 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command f2d07494 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2541 (Sat Sep 15 02:52:43 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2541 (Sat Sep 15 02:52:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ad4637d", "local_conf_timestamp": 2541, "host-ts": 2541}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2532 (Sat Sep 15 02:52:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2533 (Sat Sep 15 02:52:35 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "92d9f6e8", "local_conf_timestamp": 2533, "host-ts": 2532}, "global_maintenance": true} lago.ssh: DEBUG: start task:7221ff8d-1909-4fcf-87c8-ef62ef65ed2c:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:7221ff8d-1909-4fcf-87c8-ef62ef65ed2c:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running f544a65a on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f544a65a on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command f544a65a on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2541 (Sat Sep 15 02:52:43 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2541 (Sat Sep 15 02:52:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ad4637d", "local_conf_timestamp": 2541, "host-ts": 2541}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2532 (Sat Sep 15 02:52:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2533 (Sat Sep 15 02:52:35 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "92d9f6e8", "local_conf_timestamp": 2533, "host-ts": 2532}, "global_maintenance": true} lago.ssh: DEBUG: start task:28d5ce55-c074-4848-aef7-6a9543351375:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:28d5ce55-c074-4848-aef7-6a9543351375:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running f7b68962 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f7b68962 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command f7b68962 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2541 (Sat Sep 15 02:52:43 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2541 (Sat Sep 15 02:52:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ad4637d", "local_conf_timestamp": 2541, "host-ts": 2541}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2532 (Sat Sep 15 02:52:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2533 (Sat Sep 15 02:52:35 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "92d9f6e8", "local_conf_timestamp": 2533, "host-ts": 2532}, "global_maintenance": true} lago.ssh: DEBUG: start task:70cda777-6455-4ebe-ad9e-c50880c6f5ce:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:70cda777-6455-4ebe-ad9e-c50880c6f5ce:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running fa0f7b06 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command fa0f7b06 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command fa0f7b06 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2551 (Sat Sep 15 02:52:53 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2551 (Sat Sep 15 02:52:53 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1890ef91", "local_conf_timestamp": 2551, "host-ts": 2551}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2542 (Sat Sep 15 02:52:44 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2542 (Sat Sep 15 02:52:44 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ddd48a0d", "local_conf_timestamp": 2542, "host-ts": 2542}, "global_maintenance": true} lago.ssh: DEBUG: start task:c974937c-559f-41b0-b28b-dd0315b74d79:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:c974937c-559f-41b0-b28b-dd0315b74d79:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running fc7520bc on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command fc7520bc on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command fc7520bc on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2551 (Sat Sep 15 02:52:53 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2551 (Sat Sep 15 02:52:53 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1890ef91", "local_conf_timestamp": 2551, "host-ts": 2551}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2542 (Sat Sep 15 02:52:44 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2542 (Sat Sep 15 02:52:44 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ddd48a0d", "local_conf_timestamp": 2542, "host-ts": 2542}, "global_maintenance": true} lago.ssh: DEBUG: start task:ccf674f8-d12b-4176-a81e-8fa962e61ea7:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:ccf674f8-d12b-4176-a81e-8fa962e61ea7:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running fed087e8 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command fed087e8 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command fed087e8 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2561 (Sat Sep 15 02:53:03 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2561 (Sat Sep 15 02:53:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2e14c6de", "local_conf_timestamp": 2561, "host-ts": 2561}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2552 (Sat Sep 15 02:52:54 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2552 (Sat Sep 15 02:52:54 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host- ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7fbd8405db18> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) RuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2561 (Sat Sep 15 02:53:03 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2561 (Sat Sep 15 02:53:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2e14c6de", "local_conf_timestamp": 2561, "host-ts": 2561}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2552 (Sat Sep 15 02:52:54 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2552 (Sat Sep 15 02:52:54 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host- --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 53, in restart_he_vm _wait_for_engine_health(host) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 185, in _wait_for_engine_health testlib.assert_true_within_long(lambda: any( File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) 'could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2561 (Sat Sep 15 02:53:03 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2561 (Sat Sep 15 02:53:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2e14c6de", "local_conf_timestamp": 2561, "host-ts": 2561}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2552 (Sat Sep 15 02:52:54 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2552 (Sat Sep 15 02:52:54 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:ed47f4dc-58b7-4cd6-8718-820f742fbaa8:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:ed47f4dc-58b7-4cd6-8718-820f742fbaa8:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 9face572 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9face572 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 9face572 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2397 (Sat Sep 15 02:50:19 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2397 (Sat Sep 15 02:50:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5f097b86", "local_conf_timestamp": 2397, "host-ts": 2397}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2392 (Sat Sep 15 02:50:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2392 (Sat Sep 15 02:50:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "524cc788", "local_conf_timestamp": 2392, "host-ts": 2392}, "global_maintenance": true}\n\nroot: INFO: * Shutting down HE VM on host: lago-he-basic-ansible-suite-master-host-0\nlago.ssh: DEBUG: start task:9d2bbc93-b674-4f8f-9cff-0ed017480579:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:9d2bbc93-b674-4f8f-9cff-0ed017480579:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running a03b5ad2 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-shutdown\nlago.ssh: DEBUG: Command a03b5ad2 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be down...\nlago.ssh: DEBUG: start task:c6f14484-bc15-46ae-a11a-d71e7d7736e6:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:c6f14484-bc15-46ae-a11a-d71e7d7736e6:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running a0d7f720 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a0d7f720 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a0d7f720 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2397 (Sat Sep 15 02:50:19 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2397 (Sat Sep 15 02:50:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5f097b86", "local_conf_timestamp": 2397, "host-ts": 2397}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2392 (Sat Sep 15 02:50:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2392 (Sat Sep 15 02:50:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "524cc788", "local_conf_timestamp": 2392, "host-ts": 2392}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:b7f4b056-1ffd-4395-86d7-cbdba57b299a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:b7f4b056-1ffd-4395-86d7-cbdba57b299a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running a76cfd56 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a76cfd56 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a76cfd56 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2407 (Sat Sep 15 02:50:29 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2407 (Sat Sep 15 02:50:29 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f828e9c5", "local_conf_timestamp": 2407, "host-ts": 2407}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2402 (Sat Sep 15 02:50:24 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2402 (Sat Sep 15 02:50:25 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d36639d7", "local_conf_timestamp": 2402, "host-ts": 2402}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:0be59cd9-e4cb-4839-9891-894d2a5f611e:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:0be59cd9-e4cb-4839-9891-894d2a5f611e:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running a9bc073c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a9bc073c on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a9bc073c on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2417 (Sat Sep 15 02:50:39 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2417 (Sat Sep 15 02:50:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "42b65cd2", "local_conf_timestamp": 2417, "host-ts": 2417}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2412 (Sat Sep 15 02:50:34 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2412 (Sat Sep 15 02:50:34 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "db6596aa", "local_conf_timestamp": 2412, "host-ts": 2412}, "global_maintenance": true}\n\nroot: INFO: * VM is down.\nroot: INFO: * Stopping services...\nlago.ssh: DEBUG: start task:a7f11961-66fe-4a3e-8b5c-03640709deb3:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:a7f11961-66fe-4a3e-8b5c-03640709deb3:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running aa528b4e on lago-he-basic-ansible-suite-master-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command aa528b4e on lago-he-basic-ansible-suite-master-host-0 returned with 0\nroot: INFO: * Starting services...\nlago.ssh: DEBUG: start task:3470e088-3896-444c-9480-28f84251a111:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:3470e088-3896-444c-9480-28f84251a111:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running b42c375a on lago-he-basic-ansible-suite-master-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command b42c375a on lago-he-basic-ansible-suite-master-host-0 returned with 0\nroot: INFO: * Waiting for agent to be ready...\nlago.ssh: DEBUG: start task:c551d6c2-0baf-48c5-b907-f31f4391d1c6:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:c551d6c2-0baf-48c5-b907-f31f4391d1c6:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running b677b14c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command b677b14c on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command b677b14c on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:5dfa64c5-57dd-43f8-b09b-5e6549669059:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:5dfa64c5-57dd-43f8-b09b-5e6549669059:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running bcc603e6 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command bcc603e6 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command bcc603e6 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:4100a51d-ba55-4808-ac40-f68b20c91e05:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:4100a51d-ba55-4808-ac40-f68b20c91e05:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running beff4df2 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command beff4df2 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command beff4df2 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:46fa8f10-bbc9-433e-9b53-0029800375c7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:46fa8f10-bbc9-433e-9b53-0029800375c7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running c127b556 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command c127b556 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command c127b556 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:d5c1c51b-f487-4b59-8079-bdb59bdfcfa7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:d5c1c51b-f487-4b59-8079-bdb59bdfcfa7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running c360ebd0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command c360ebd0 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command c360ebd0 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:2dbd6214-28f6-4736-9a97-0e98076457c1:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:2dbd6214-28f6-4736-9a97-0e98076457c1:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running c58e6b3a on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command c58e6b3a on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command c58e6b3a on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:502b040a-af3b-4f44-aed9-9dbce56933d2:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:502b040a-af3b-4f44-aed9-9dbce56933d2:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running c7b6c876 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command c7b6c876 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command c7b6c876 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:51e6fde8-4042-4144-930e-252299ee405c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:51e6fde8-4042-4144-930e-252299ee405c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running c9e638c0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command c9e638c0 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command c9e638c0 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:508cff8b-8c2a-4584-b493-d2679f10ad82:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:508cff8b-8c2a-4584-b493-d2679f10ad82:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running cc0c3816 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command cc0c3816 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command cc0c3816 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:6033b3fb-f21b-447d-a137-04e7784aab4a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:6033b3fb-f21b-447d-a137-04e7784aab4a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running ce4de886 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command ce4de886 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ce4de886 on lago-he-basic-ansible-suite-master-host-0 output:\n \n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\n\n--== Host lago-he-basic-ansible-suite-master-host-0 (id: 1) status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : True\nHostname : lago-he-basic-ansible-suite-master-host-0\nHost ID : 1\nEngine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}\nScore : 0\nstopped : False\nLocal maintenance : False\ncrc32 : 1bfd8aa6\nlocal_conf_timestamp : 2481\nHost timestamp : 2481\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2481 (Sat Sep 15 02:51:43 2018)\n\thost-id=1\n\tscore=0\n\tvm_conf_refresh_time=2481 (Sat Sep 15 02:51:43 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=ReinitializeFSM\n\tstopped=False\n\n\n--== Host lago-he-basic-ansible-suite-master-host-1 (id: 2) status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : False\nHostname : lago-he-basic-ansible-suite-master-host-1\nHost ID : 2\nEngine status : unknown stale-data\nScore : 3400\nstopped : False\nLocal maintenance : False\ncrc32 : eb6c75a4\nlocal_conf_timestamp : 2472\nHost timestamp : 2472\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2472 (Sat Sep 15 02:51:34 2018)\n\thost-id=2\n\tscore=3400\n\tvm_conf_refresh_time=2472 (Sat Sep 15 02:51:34 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=GlobalMaintenance\n\tstopped=False\n\n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\nroot: INFO: * Agent is ready.\nroot: INFO: * Starting VM...\nlago.ssh: DEBUG: start task:4b642e15-e2d0-441d-b6c7-6726e4671726:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:4b642e15-e2d0-441d-b6c7-6726e4671726:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running cecfa204 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-start\nlago.ssh: DEBUG: Command cecfa204 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command cecfa204 on lago-he-basic-ansible-suite-master-host-0 output:\n VM exists and is down, cleaning up and restarting\n\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be UP...\nlago.ssh: DEBUG: start task:46acaf72-3d13-4871-abd2-af9b3e9be8e0:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:46acaf72-3d13-4871-abd2-af9b3e9be8e0:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running d022e580 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command d022e580 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command d022e580 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2481 (Sat Sep 15 02:51:43 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=2481 (Sat Sep 15 02:51:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=ReinitializeFSM\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "1bfd8aa6", "local_conf_timestamp": 2481, "host-ts": 2481}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2472 (Sat Sep 15 02:51:34 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2472 (Sat Sep 15 02:51:34 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eb6c75a4", "local_conf_timestamp": 2472, "host-ts": 2472}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:9aacbec2-ebe2-4adb-980c-28909ccec04a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:9aacbec2-ebe2-4adb-980c-28909ccec04a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running d698d1ea on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command d698d1ea on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command d698d1ea on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2491 (Sat Sep 15 02:51:53 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2491 (Sat Sep 15 02:51:53 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e4fd4d62", "local_conf_timestamp": 2491, "host-ts": 2491}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2482 (Sat Sep 15 02:51:44 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2482 (Sat Sep 15 02:51:45 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "937b423f", "local_conf_timestamp": 2482, "host-ts": 2482}, "global_maintenance": true}\n\nroot: INFO: * VM is UP.\nroot: INFO: * Waiting for engine to start...\nlago.ssh: DEBUG: start task:a1f19e92-cf1d-461b-8dfe-54f39840068c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:a1f19e92-cf1d-461b-8dfe-54f39840068c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running d7183cd2 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command d7183cd2 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command d7183cd2 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2491 (Sat Sep 15 02:51:53 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2491 (Sat Sep 15 02:51:53 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e4fd4d62", "local_conf_timestamp": 2491, "host-ts": 2491}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2482 (Sat Sep 15 02:51:44 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2482 (Sat Sep 15 02:51:45 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "937b423f", "local_conf_timestamp": 2482, "host-ts": 2482}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:7b048778-3a8f-468f-a3ba-40adba395ce2:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:7b048778-3a8f-468f-a3ba-40adba395ce2:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running dd98827e on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command dd98827e on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command dd98827e on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2501 (Sat Sep 15 02:52:03 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2502 (Sat Sep 15 02:52:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd877c52", "local_conf_timestamp": 2502, "host-ts": 2501}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2492 (Sat Sep 15 02:51:55 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2493 (Sat Sep 15 02:51:55 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ec7fddd4", "local_conf_timestamp": 2493, "host-ts": 2492}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:dec50117-7911-436c-92ab-23a6e01b9797:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:dec50117-7911-436c-92ab-23a6e01b9797:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running dfee1836 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command dfee1836 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command dfee1836 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2511 (Sat Sep 15 02:52:13 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2511 (Sat Sep 15 02:52:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c163b966", "local_conf_timestamp": 2511, "host-ts": 2511}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2492 (Sat Sep 15 02:51:55 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2493 (Sat Sep 15 02:51:55 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ec7fddd4", "local_conf_timestamp": 2493, "host-ts": 2492}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:3d54d54a-3318-45ce-9635-fa7c36c532e9:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:3d54d54a-3318-45ce-9635-fa7c36c532e9:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running e24c7e9c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e24c7e9c on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command e24c7e9c on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2511 (Sat Sep 15 02:52:13 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2511 (Sat Sep 15 02:52:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c163b966", "local_conf_timestamp": 2511, "host-ts": 2511}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2502 (Sat Sep 15 02:52:04 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2502 (Sat Sep 15 02:52:04 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fdda37f9", "local_conf_timestamp": 2502, "host-ts": 2502}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:081e05c7-1491-4111-a828-dcdfb7750adb:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:081e05c7-1491-4111-a828-dcdfb7750adb:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running e4a9c406 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e4a9c406 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command e4a9c406 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2511 (Sat Sep 15 02:52:13 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2511 (Sat Sep 15 02:52:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c163b966", "local_conf_timestamp": 2511, "host-ts": 2511}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2502 (Sat Sep 15 02:52:04 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2502 (Sat Sep 15 02:52:04 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fdda37f9", "local_conf_timestamp": 2502, "host-ts": 2502}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ed1622e6-c317-4d65-878b-c034ae4c684a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:ed1622e6-c317-4d65-878b-c034ae4c684a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running e72300d0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e72300d0 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command e72300d0 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2521 (Sat Sep 15 02:52:23 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2521 (Sat Sep 15 02:52:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "77dc31e3", "local_conf_timestamp": 2521, "host-ts": 2521}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2512 (Sat Sep 15 02:52:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2512 (Sat Sep 15 02:52:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f5d99884", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:b0e31724-b7d5-4e10-ba19-b85cd6a18237:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:b0e31724-b7d5-4e10-ba19-b85cd6a18237:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running e9946412 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e9946412 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command e9946412 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2521 (Sat Sep 15 02:52:23 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2521 (Sat Sep 15 02:52:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "77dc31e3", "local_conf_timestamp": 2521, "host-ts": 2521}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2512 (Sat Sep 15 02:52:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2512 (Sat Sep 15 02:52:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f5d99884", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:a3e7508c-a4ab-4711-94c6-9274d4dd866a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:a3e7508c-a4ab-4711-94c6-9274d4dd866a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running ebdc77f0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ebdc77f0 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ebdc77f0 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2521 (Sat Sep 15 02:52:23 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2521 (Sat Sep 15 02:52:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "77dc31e3", "local_conf_timestamp": 2521, "host-ts": 2521}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2512 (Sat Sep 15 02:52:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2512 (Sat Sep 15 02:52:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f5d99884", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:07a8ca1d-6dae-49f5-9a41-8ff5113c3330:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:07a8ca1d-6dae-49f5-9a41-8ff5113c3330:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running ee2ea7e4 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ee2ea7e4 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ee2ea7e4 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2531 (Sat Sep 15 02:52:33 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2531 (Sat Sep 15 02:52:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "65a02926", "local_conf_timestamp": 2531, "host-ts": 2531}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2522 (Sat Sep 15 02:52:24 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2522 (Sat Sep 15 02:52:24 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eddd6903", "local_conf_timestamp": 2522, "host-ts": 2522}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ff262ca1-ab96-4d86-b032-6684126b2551:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:ff262ca1-ab96-4d86-b032-6684126b2551:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running f06d4998 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f06d4998 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command f06d4998 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2531 (Sat Sep 15 02:52:33 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2531 (Sat Sep 15 02:52:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "65a02926", "local_conf_timestamp": 2531, "host-ts": 2531}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2522 (Sat Sep 15 02:52:24 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2522 (Sat Sep 15 02:52:24 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eddd6903", "local_conf_timestamp": 2522, "host-ts": 2522}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:0bd6fcfe-a362-4cbb-9030-45e36f7121a3:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:0bd6fcfe-a362-4cbb-9030-45e36f7121a3:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running f2d07494 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f2d07494 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command f2d07494 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2541 (Sat Sep 15 02:52:43 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2541 (Sat Sep 15 02:52:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ad4637d", "local_conf_timestamp": 2541, "host-ts": 2541}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2532 (Sat Sep 15 02:52:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2533 (Sat Sep 15 02:52:35 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "92d9f6e8", "local_conf_timestamp": 2533, "host-ts": 2532}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:7221ff8d-1909-4fcf-87c8-ef62ef65ed2c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:7221ff8d-1909-4fcf-87c8-ef62ef65ed2c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running f544a65a on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f544a65a on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command f544a65a on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2541 (Sat Sep 15 02:52:43 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2541 (Sat Sep 15 02:52:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ad4637d", "local_conf_timestamp": 2541, "host-ts": 2541}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2532 (Sat Sep 15 02:52:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2533 (Sat Sep 15 02:52:35 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "92d9f6e8", "local_conf_timestamp": 2533, "host-ts": 2532}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:28d5ce55-c074-4848-aef7-6a9543351375:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:28d5ce55-c074-4848-aef7-6a9543351375:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running f7b68962 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f7b68962 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command f7b68962 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2541 (Sat Sep 15 02:52:43 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2541 (Sat Sep 15 02:52:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ad4637d", "local_conf_timestamp": 2541, "host-ts": 2541}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2532 (Sat Sep 15 02:52:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2533 (Sat Sep 15 02:52:35 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "92d9f6e8", "local_conf_timestamp": 2533, "host-ts": 2532}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:70cda777-6455-4ebe-ad9e-c50880c6f5ce:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:70cda777-6455-4ebe-ad9e-c50880c6f5ce:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running fa0f7b06 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fa0f7b06 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command fa0f7b06 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2551 (Sat Sep 15 02:52:53 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2551 (Sat Sep 15 02:52:53 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1890ef91", "local_conf_timestamp": 2551, "host-ts": 2551}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2542 (Sat Sep 15 02:52:44 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2542 (Sat Sep 15 02:52:44 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ddd48a0d", "local_conf_timestamp": 2542, "host-ts": 2542}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:c974937c-559f-41b0-b28b-dd0315b74d79:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:c974937c-559f-41b0-b28b-dd0315b74d79:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running fc7520bc on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fc7520bc on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command fc7520bc on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2551 (Sat Sep 15 02:52:53 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2551 (Sat Sep 15 02:52:53 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1890ef91", "local_conf_timestamp": 2551, "host-ts": 2551}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2542 (Sat Sep 15 02:52:44 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2542 (Sat Sep 15 02:52:44 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ddd48a0d", "local_conf_timestamp": 2542, "host-ts": 2542}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ccf674f8-d12b-4176-a81e-8fa962e61ea7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:ccf674f8-d12b-4176-a81e-8fa962e61ea7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running fed087e8 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fed087e8 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command fed087e8 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2561 (Sat Sep 15 02:53:03 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2561 (Sat Sep 15 02:53:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2e14c6de", "local_conf_timestamp": 2561, "host-ts": 2561}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2552 (Sat Sep 15 02:52:54 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2552 (Sat Sep 15 02:52:54 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7fbd8405db18>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda>\n for k, v in _get_he_status(host).items()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status\n raise RuntimeError(\'could not parse JSON: %s\' % ret.out)\nRuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2561 (Sat Sep 15 02:53:03 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2561 (Sat Sep 15 02:53:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2e14c6de", "local_conf_timestamp": 2561, "host-ts": 2561}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2552 (Sat Sep 15 02:52:54 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2552 (Sat Sep 15 02:52:54 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-\n--------------------- >> end captured logging << ---------------------'

1 1

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #1946
by jenkins＠jenkins.phx.ovirt.org 15 Sep '18

15 Sep '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 9c187d6e9145db35423f495b553304ceb75489cb (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 9c187d6e9145db35423f495b553304ceb75489cb Commit message: "Add jobs for ovs-cni" > git rev-list --no-walk 9c187d6e9145db35423f495b553304ceb75489cb # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins4484725594797596479.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7/packages error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7/packages' Build step 'Execute shell' marked build as failure

1 1

[CQ]: 94317,2 (vdsm) failed "ovirt-master" system tests
by oVirt Jenkins 15 Sep '18

15 Sep '18

Change 94317,2 (vdsm) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/94317/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10323/

1 0

oVirt infra daily report - unstable production jobs - 583
by jenkins＠jenkins.phx.ovirt.org 15 Sep '18

15 Sep '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/583//artifact/exported-a… Cheers, Jenkins

1 0

[CQ]: 94307, 6 (ovirt-provider-ovn) failed "ovirt-master" system tests
by oVirt Jenkins 14 Sep '18

14 Sep '18

Change 94307,6 (ovirt-provider-ovn) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/94307/6 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10318/

1 0