- Infra - oVirt List Archives

[oVirt Jenkins] ovirt-system-tests_network-suite-4.2 - Build # 581 - Failure!
by jenkins＠jenkins.phx.ovirt.org 16 Sep '18

16 Sep '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/581/ Build Number: 581 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #581 [Milan Zamazal] Update test-vm.ovf file ----------------- Failed Tests: ----------------- No tests ran.

1 1

[CQ]: 94337, 2 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 16 Sep '18

16 Sep '18

A system test invoked by the "ovirt-master" change queue including change 94337,2 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 94081,3 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 94081,3 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/94337/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/94081/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10326/

1 0

[JIRA] (OVIRT-2498) Failing KubeVirt CI
by Barak Korren (oVirt JIRA) 16 Sep '18

16 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2498?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-2498: ------------------------------------- [~pkotas] I think the best place to discuss Kubevirt issues is on the Kubevirt-related mailing lists where other Kubvirt developers can see the discussion. To your questions: {quote} I am working on fixing the issues on the KubeVirt e2e test suites. This task is directly related to unstable CI, due to unknown errors. The progress is reported in the CNV trello: https://trello.com/c/HNXcMEQu/161-epic-improve-ci I am creating this issue since the KubeVirt experience random timeouts on random tests most of the times when test suites run. The issue from outside is showing as timeouts on difference part of tests. Sometimes the tests fails in set up phase, again due to random timeout. The example in the link bellow timed out for network connection on localhost. [check-patch.k8s-1.11.0-dev.el7.x86_64] requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60) {quote} Its generally a bad idea to rely to much on timeout in a test suit like this. We've seen such issues over and over again in OST as well. Its probably best to remove all such timeout definitions and just have overall timeout set for the entire test suit. {quote} Example of failing test suites is here https://jenkins.ovirt.org/job/kubevirt_kubevirt_standard-check-pr/1916/cons… The list of errors related to the failing CI can be found in my notes https://docs.google.com/document/d/1_ll1DOMHgCRHn_Df9i4uvtRFyMK-bDCHEeGfJFT… I am not sure whether KubeVirt already shared the resource requirements, so I provide short summary: Resources for KubeVirt e2e tests: at least 12GB of RAM - we start 3 nodes (3 docker images) each require 4GB of RAM exposed /dev/kvm to enable native virtualization cached images, since these are used to build the test cluster: kubevirtci/os-3.10.0-crio:latest kubevirtci/os-3.10.0-multus:latest kubevirtci/os-3.10.0:latest kubevirtci/k8s-1.10.4:latest kubevirtci/k8s-multus-1.11.1:latest kubevirtci/k8s-1.11.0:latest How can we overcome this? Can we work together to build a suitable requirements for running the tests so it passes each time? {quote} To my knowledge the existing setup meets all the requirements you specify above. We have 3 physical hosts that are used to run Kubevirt tests, each host has 128GB of ram and runs 7 containers where each container runs its own Libvirt, Docker and Systemd so that it looks like its own host to the tests running inside. The amount of containers per host was calculated to have each container have a little over 16GB of RAM for itself. So we should have more then enough for Kubevirt. Also, in our measurements Kubevirt's CI tests took way less then 12GB, ansd were somewhere around 8GB. All the images that start with 'kubevirtci' are cached by the system. WRT /dev/kvm - we do have it exposed in the containers we run, but I think that is irrelevant since AFAIK Kubevirt-CI runs qemu on its own inside its own container, so the /dev/kvm device files simply needs to exist inside that container. > Failing KubeVirt CI > ------------------- > > Key: OVIRT-2498 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2498 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Petr Kotas > Assignee: infra > > Hi, > I am working on fixing the issues on the KubeVirt e2e test suites. This > task is directly related to unstable CI, due to unknown errors. > The progress is reported in the CNV trello: > https://trello.com/c/HNXcMEQu/161-epic-improve-ci > I am creating this issue since the KubeVirt experience random timeouts on > random tests most of the times when test suites run. > The issue from outside is showing as timeouts on difference part of tests. > Sometimes the tests fails in set up phase, again due to random timeout. > The example in the link bellow timed out for network connection on > localhost. > [check-patch.k8s-1.11.0-dev.el7.x86_64] > requests.exceptions.ReadTimeout: > UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. > (read timeout=60) > Example of failing test suites is here > https://jenkins.ovirt.org/job/kubevirt_kubevirt_standard-check-pr/1916/cons… > The list of errors related to the failing CI can be found in my notes > https://docs.google.com/document/d/1_ll1DOMHgCRHn_Df9i4uvtRFyMK-bDCHEeGfJFT… > I am not sure whether KubeVirt already shared the resource requirements, so > I provide short summary: > *Resources for KubeVirt e2e tests:* > - at least 12GB of RAM - we start 3 nodes (3 docker images) each require > 4GB of RAM > - exposed /dev/kvm to enable native virtualization > - cached images, since these are used to build the test cluster: > - kubevirtci/os-3.10.0-crio:latest > - kubevirtci/os-3.10.0-multus:latest > - kubevirtci/os-3.10.0:latest > - kubevirtci/k8s-1.10.4:latest > - kubevirtci/k8s-multus-1.11.1:latest > - kubevirtci/k8s-1.11.0:latest > How can we overcome this? Can we work together to build a suitable > requirements for running the tests so it passes each time? > Kind regards, > Petr Kotas -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

oVirt infra daily report - unstable production jobs - 584
by jenkins＠jenkins.phx.ovirt.org 16 Sep '18

16 Sep '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/584//artifact/exported-a… Cheers, Jenkins

1 0

[oVirt Jenkins] ovirt-system-tests_he-basic-ansible-suite-master - Build # 632 - Failure!
by jenkins＠jenkins.phx.ovirt.org 15 Sep '18

15 Sep '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-mast… Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-mast… Build Number: 632 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #632 [Milan Zamazal] Update test-vm.ovf file ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 008_restart_he_vm.restart_he_vm Error Message: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2561 (Sat Sep 15 02:53:03 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2561 (Sat Sep 15 02:53:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2e14c6de", "local_conf_timestamp": 2561, "host-ts": 2561}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2552 (Sat Sep 15 02:52:54 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2552 (Sat Sep 15 02:52:54 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host- -------------------- >> begin captured logging << -------------------- lago.ssh: DEBUG: start task:ed47f4dc-58b7-4cd6-8718-820f742fbaa8:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:ed47f4dc-58b7-4cd6-8718-820f742fbaa8:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 9face572 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9face572 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 9face572 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2397 (Sat Sep 15 02:50:19 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2397 (Sat Sep 15 02:50:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5f097b86", "local_conf_timestamp": 2397, "host-ts": 2397}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2392 (Sat Sep 15 02:50:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2392 (Sat Sep 15 02:50:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "524cc788", "local_conf_timestamp": 2392, "host-ts": 2392}, "global_maintenance": true} root: INFO: * Shutting down HE VM on host: lago-he-basic-ansible-suite-master-host-0 lago.ssh: DEBUG: start task:9d2bbc93-b674-4f8f-9cff-0ed017480579:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:9d2bbc93-b674-4f8f-9cff-0ed017480579:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running a03b5ad2 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-shutdown lago.ssh: DEBUG: Command a03b5ad2 on lago-he-basic-ansible-suite-master-host-0 returned with 0 root: INFO: * Command succeeded root: INFO: * Waiting for VM to be down... lago.ssh: DEBUG: start task:c6f14484-bc15-46ae-a11a-d71e7d7736e6:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:c6f14484-bc15-46ae-a11a-d71e7d7736e6:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running a0d7f720 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a0d7f720 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command a0d7f720 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2397 (Sat Sep 15 02:50:19 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2397 (Sat Sep 15 02:50:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5f097b86", "local_conf_timestamp": 2397, "host-ts": 2397}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2392 (Sat Sep 15 02:50:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2392 (Sat Sep 15 02:50:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "524cc788", "local_conf_timestamp": 2392, "host-ts": 2392}, "global_maintenance": true} lago.ssh: DEBUG: start task:b7f4b056-1ffd-4395-86d7-cbdba57b299a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:b7f4b056-1ffd-4395-86d7-cbdba57b299a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running a76cfd56 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a76cfd56 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command a76cfd56 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2407 (Sat Sep 15 02:50:29 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2407 (Sat Sep 15 02:50:29 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f828e9c5", "local_conf_timestamp": 2407, "host-ts": 2407}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2402 (Sat Sep 15 02:50:24 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2402 (Sat Sep 15 02:50:25 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d36639d7", "local_conf_timestamp": 2402, "host-ts": 2402}, "global_maintenance": true} lago.ssh: DEBUG: start task:0be59cd9-e4cb-4839-9891-894d2a5f611e:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:0be59cd9-e4cb-4839-9891-894d2a5f611e:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running a9bc073c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a9bc073c on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command a9bc073c on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2417 (Sat Sep 15 02:50:39 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2417 (Sat Sep 15 02:50:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "42b65cd2", "local_conf_timestamp": 2417, "host-ts": 2417}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2412 (Sat Sep 15 02:50:34 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2412 (Sat Sep 15 02:50:34 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "db6596aa", "local_conf_timestamp": 2412, "host-ts": 2412}, "global_maintenance": true} root: INFO: * VM is down. root: INFO: * Stopping services... lago.ssh: DEBUG: start task:a7f11961-66fe-4a3e-8b5c-03640709deb3:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:a7f11961-66fe-4a3e-8b5c-03640709deb3:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running aa528b4e on lago-he-basic-ansible-suite-master-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command aa528b4e on lago-he-basic-ansible-suite-master-host-0 returned with 0 root: INFO: * Starting services... lago.ssh: DEBUG: start task:3470e088-3896-444c-9480-28f84251a111:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:3470e088-3896-444c-9480-28f84251a111:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running b42c375a on lago-he-basic-ansible-suite-master-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command b42c375a on lago-he-basic-ansible-suite-master-host-0 returned with 0 root: INFO: * Waiting for agent to be ready... lago.ssh: DEBUG: start task:c551d6c2-0baf-48c5-b907-f31f4391d1c6:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:c551d6c2-0baf-48c5-b907-f31f4391d1c6:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running b677b14c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command b677b14c on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command b677b14c on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:5dfa64c5-57dd-43f8-b09b-5e6549669059:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:5dfa64c5-57dd-43f8-b09b-5e6549669059:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running bcc603e6 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command bcc603e6 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command bcc603e6 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:4100a51d-ba55-4808-ac40-f68b20c91e05:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:4100a51d-ba55-4808-ac40-f68b20c91e05:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running beff4df2 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command beff4df2 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command beff4df2 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:46fa8f10-bbc9-433e-9b53-0029800375c7:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:46fa8f10-bbc9-433e-9b53-0029800375c7:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running c127b556 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command c127b556 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command c127b556 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:d5c1c51b-f487-4b59-8079-bdb59bdfcfa7:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:d5c1c51b-f487-4b59-8079-bdb59bdfcfa7:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running c360ebd0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command c360ebd0 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command c360ebd0 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:2dbd6214-28f6-4736-9a97-0e98076457c1:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:2dbd6214-28f6-4736-9a97-0e98076457c1:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running c58e6b3a on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command c58e6b3a on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command c58e6b3a on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:502b040a-af3b-4f44-aed9-9dbce56933d2:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:502b040a-af3b-4f44-aed9-9dbce56933d2:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running c7b6c876 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command c7b6c876 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command c7b6c876 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:51e6fde8-4042-4144-930e-252299ee405c:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:51e6fde8-4042-4144-930e-252299ee405c:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running c9e638c0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command c9e638c0 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command c9e638c0 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:508cff8b-8c2a-4584-b493-d2679f10ad82:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:508cff8b-8c2a-4584-b493-d2679f10ad82:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running cc0c3816 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command cc0c3816 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command cc0c3816 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:6033b3fb-f21b-447d-a137-04e7784aab4a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:6033b3fb-f21b-447d-a137-04e7784aab4a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running ce4de886 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command ce4de886 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command ce4de886 on lago-he-basic-ansible-suite-master-host-0 output: !! Cluster is in GLOBAL MAINTENANCE mode !! --== Host lago-he-basic-ansible-suite-master-host-0 (id: 1) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : lago-he-basic-ansible-suite-master-host-0 Host ID : 1 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"} Score : 0 stopped : False Local maintenance : False crc32 : 1bfd8aa6 local_conf_timestamp : 2481 Host timestamp : 2481 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2481 (Sat Sep 15 02:51:43 2018) host-id=1 score=0 vm_conf_refresh_time=2481 (Sat Sep 15 02:51:43 2018) conf_on_shared_storage=True maintenance=False state=ReinitializeFSM stopped=False --== Host lago-he-basic-ansible-suite-master-host-1 (id: 2) status ==-- conf_on_shared_storage : True Status up-to-date : False Hostname : lago-he-basic-ansible-suite-master-host-1 Host ID : 2 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : eb6c75a4 local_conf_timestamp : 2472 Host timestamp : 2472 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2472 (Sat Sep 15 02:51:34 2018) host-id=2 score=3400 vm_conf_refresh_time=2472 (Sat Sep 15 02:51:34 2018) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False !! Cluster is in GLOBAL MAINTENANCE mode !! root: INFO: * Agent is ready. root: INFO: * Starting VM... lago.ssh: DEBUG: start task:4b642e15-e2d0-441d-b6c7-6726e4671726:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:4b642e15-e2d0-441d-b6c7-6726e4671726:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running cecfa204 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-start lago.ssh: DEBUG: Command cecfa204 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command cecfa204 on lago-he-basic-ansible-suite-master-host-0 output: VM exists and is down, cleaning up and restarting root: INFO: * Command succeeded root: INFO: * Waiting for VM to be UP... lago.ssh: DEBUG: start task:46acaf72-3d13-4871-abd2-af9b3e9be8e0:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:46acaf72-3d13-4871-abd2-af9b3e9be8e0:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running d022e580 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command d022e580 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command d022e580 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2481 (Sat Sep 15 02:51:43 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=2481 (Sat Sep 15 02:51:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=ReinitializeFSM\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "1bfd8aa6", "local_conf_timestamp": 2481, "host-ts": 2481}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2472 (Sat Sep 15 02:51:34 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2472 (Sat Sep 15 02:51:34 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eb6c75a4", "local_conf_timestamp": 2472, "host-ts": 2472}, "global_maintenance": true} lago.ssh: DEBUG: start task:9aacbec2-ebe2-4adb-980c-28909ccec04a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:9aacbec2-ebe2-4adb-980c-28909ccec04a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running d698d1ea on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command d698d1ea on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command d698d1ea on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2491 (Sat Sep 15 02:51:53 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2491 (Sat Sep 15 02:51:53 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e4fd4d62", "local_conf_timestamp": 2491, "host-ts": 2491}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2482 (Sat Sep 15 02:51:44 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2482 (Sat Sep 15 02:51:45 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "937b423f", "local_conf_timestamp": 2482, "host-ts": 2482}, "global_maintenance": true} root: INFO: * VM is UP. root: INFO: * Waiting for engine to start... lago.ssh: DEBUG: start task:a1f19e92-cf1d-461b-8dfe-54f39840068c:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:a1f19e92-cf1d-461b-8dfe-54f39840068c:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running d7183cd2 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command d7183cd2 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command d7183cd2 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2491 (Sat Sep 15 02:51:53 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2491 (Sat Sep 15 02:51:53 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e4fd4d62", "local_conf_timestamp": 2491, "host-ts": 2491}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2482 (Sat Sep 15 02:51:44 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2482 (Sat Sep 15 02:51:45 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "937b423f", "local_conf_timestamp": 2482, "host-ts": 2482}, "global_maintenance": true} lago.ssh: DEBUG: start task:7b048778-3a8f-468f-a3ba-40adba395ce2:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:7b048778-3a8f-468f-a3ba-40adba395ce2:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running dd98827e on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command dd98827e on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command dd98827e on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2501 (Sat Sep 15 02:52:03 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2502 (Sat Sep 15 02:52:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd877c52", "local_conf_timestamp": 2502, "host-ts": 2501}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2492 (Sat Sep 15 02:51:55 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2493 (Sat Sep 15 02:51:55 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ec7fddd4", "local_conf_timestamp": 2493, "host-ts": 2492}, "global_maintenance": true} lago.ssh: DEBUG: start task:dec50117-7911-436c-92ab-23a6e01b9797:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:dec50117-7911-436c-92ab-23a6e01b9797:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running dfee1836 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command dfee1836 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command dfee1836 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2511 (Sat Sep 15 02:52:13 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2511 (Sat Sep 15 02:52:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c163b966", "local_conf_timestamp": 2511, "host-ts": 2511}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2492 (Sat Sep 15 02:51:55 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2493 (Sat Sep 15 02:51:55 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ec7fddd4", "local_conf_timestamp": 2493, "host-ts": 2492}, "global_maintenance": true} lago.ssh: DEBUG: start task:3d54d54a-3318-45ce-9635-fa7c36c532e9:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:3d54d54a-3318-45ce-9635-fa7c36c532e9:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running e24c7e9c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e24c7e9c on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command e24c7e9c on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2511 (Sat Sep 15 02:52:13 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2511 (Sat Sep 15 02:52:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c163b966", "local_conf_timestamp": 2511, "host-ts": 2511}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2502 (Sat Sep 15 02:52:04 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2502 (Sat Sep 15 02:52:04 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fdda37f9", "local_conf_timestamp": 2502, "host-ts": 2502}, "global_maintenance": true} lago.ssh: DEBUG: start task:081e05c7-1491-4111-a828-dcdfb7750adb:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:081e05c7-1491-4111-a828-dcdfb7750adb:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running e4a9c406 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e4a9c406 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command e4a9c406 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2511 (Sat Sep 15 02:52:13 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2511 (Sat Sep 15 02:52:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c163b966", "local_conf_timestamp": 2511, "host-ts": 2511}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2502 (Sat Sep 15 02:52:04 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2502 (Sat Sep 15 02:52:04 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fdda37f9", "local_conf_timestamp": 2502, "host-ts": 2502}, "global_maintenance": true} lago.ssh: DEBUG: start task:ed1622e6-c317-4d65-878b-c034ae4c684a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:ed1622e6-c317-4d65-878b-c034ae4c684a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running e72300d0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e72300d0 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command e72300d0 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2521 (Sat Sep 15 02:52:23 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2521 (Sat Sep 15 02:52:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "77dc31e3", "local_conf_timestamp": 2521, "host-ts": 2521}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2512 (Sat Sep 15 02:52:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2512 (Sat Sep 15 02:52:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f5d99884", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true} lago.ssh: DEBUG: start task:b0e31724-b7d5-4e10-ba19-b85cd6a18237:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:b0e31724-b7d5-4e10-ba19-b85cd6a18237:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running e9946412 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e9946412 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command e9946412 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2521 (Sat Sep 15 02:52:23 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2521 (Sat Sep 15 02:52:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "77dc31e3", "local_conf_timestamp": 2521, "host-ts": 2521}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2512 (Sat Sep 15 02:52:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2512 (Sat Sep 15 02:52:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f5d99884", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true} lago.ssh: DEBUG: start task:a3e7508c-a4ab-4711-94c6-9274d4dd866a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:a3e7508c-a4ab-4711-94c6-9274d4dd866a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running ebdc77f0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ebdc77f0 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command ebdc77f0 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2521 (Sat Sep 15 02:52:23 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2521 (Sat Sep 15 02:52:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "77dc31e3", "local_conf_timestamp": 2521, "host-ts": 2521}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2512 (Sat Sep 15 02:52:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2512 (Sat Sep 15 02:52:14 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f5d99884", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true} lago.ssh: DEBUG: start task:07a8ca1d-6dae-49f5-9a41-8ff5113c3330:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:07a8ca1d-6dae-49f5-9a41-8ff5113c3330:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running ee2ea7e4 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ee2ea7e4 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command ee2ea7e4 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2531 (Sat Sep 15 02:52:33 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2531 (Sat Sep 15 02:52:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "65a02926", "local_conf_timestamp": 2531, "host-ts": 2531}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2522 (Sat Sep 15 02:52:24 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2522 (Sat Sep 15 02:52:24 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eddd6903", "local_conf_timestamp": 2522, "host-ts": 2522}, "global_maintenance": true} lago.ssh: DEBUG: start task:ff262ca1-ab96-4d86-b032-6684126b2551:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:ff262ca1-ab96-4d86-b032-6684126b2551:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running f06d4998 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f06d4998 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command f06d4998 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2531 (Sat Sep 15 02:52:33 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2531 (Sat Sep 15 02:52:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "65a02926", "local_conf_timestamp": 2531, "host-ts": 2531}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2522 (Sat Sep 15 02:52:24 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2522 (Sat Sep 15 02:52:24 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eddd6903", "local_conf_timestamp": 2522, "host-ts": 2522}, "global_maintenance": true} lago.ssh: DEBUG: start task:0bd6fcfe-a362-4cbb-9030-45e36f7121a3:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:0bd6fcfe-a362-4cbb-9030-45e36f7121a3:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running f2d07494 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f2d07494 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command f2d07494 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2541 (Sat Sep 15 02:52:43 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2541 (Sat Sep 15 02:52:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ad4637d", "local_conf_timestamp": 2541, "host-ts": 2541}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2532 (Sat Sep 15 02:52:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2533 (Sat Sep 15 02:52:35 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "92d9f6e8", "local_conf_timestamp": 2533, "host-ts": 2532}, "global_maintenance": true} lago.ssh: DEBUG: start task:7221ff8d-1909-4fcf-87c8-ef62ef65ed2c:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:7221ff8d-1909-4fcf-87c8-ef62ef65ed2c:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running f544a65a on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f544a65a on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command f544a65a on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2541 (Sat Sep 15 02:52:43 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2541 (Sat Sep 15 02:52:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ad4637d", "local_conf_timestamp": 2541, "host-ts": 2541}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2532 (Sat Sep 15 02:52:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2533 (Sat Sep 15 02:52:35 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "92d9f6e8", "local_conf_timestamp": 2533, "host-ts": 2532}, "global_maintenance": true} lago.ssh: DEBUG: start task:28d5ce55-c074-4848-aef7-6a9543351375:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:28d5ce55-c074-4848-aef7-6a9543351375:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running f7b68962 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f7b68962 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command f7b68962 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2541 (Sat Sep 15 02:52:43 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2541 (Sat Sep 15 02:52:43 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ad4637d", "local_conf_timestamp": 2541, "host-ts": 2541}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2532 (Sat Sep 15 02:52:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2533 (Sat Sep 15 02:52:35 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "92d9f6e8", "local_conf_timestamp": 2533, "host-ts": 2532}, "global_maintenance": true} lago.ssh: DEBUG: start task:70cda777-6455-4ebe-ad9e-c50880c6f5ce:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:70cda777-6455-4ebe-ad9e-c50880c6f5ce:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running fa0f7b06 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command fa0f7b06 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command fa0f7b06 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2551 (Sat Sep 15 02:52:53 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2551 (Sat Sep 15 02:52:53 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1890ef91", "local_conf_timestamp": 2551, "host-ts": 2551}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2542 (Sat Sep 15 02:52:44 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2542 (Sat Sep 15 02:52:44 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ddd48a0d", "local_conf_timestamp": 2542, "host-ts": 2542}, "global_maintenance": true} lago.ssh: DEBUG: start task:c974937c-559f-41b0-b28b-dd0315b74d79:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:c974937c-559f-41b0-b28b-dd0315b74d79:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running fc7520bc on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command fc7520bc on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command fc7520bc on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2551 (Sat Sep 15 02:52:53 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2551 (Sat Sep 15 02:52:53 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1890ef91", "local_conf_timestamp": 2551, "host-ts": 2551}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2542 (Sat Sep 15 02:52:44 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2542 (Sat Sep 15 02:52:44 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ddd48a0d", "local_conf_timestamp": 2542, "host-ts": 2542}, "global_maintenance": true} lago.ssh: DEBUG: start task:ccf674f8-d12b-4176-a81e-8fa962e61ea7:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:ccf674f8-d12b-4176-a81e-8fa962e61ea7:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running fed087e8 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command fed087e8 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command fed087e8 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2561 (Sat Sep 15 02:53:03 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2561 (Sat Sep 15 02:53:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2e14c6de", "local_conf_timestamp": 2561, "host-ts": 2561}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2552 (Sat Sep 15 02:52:54 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2552 (Sat Sep 15 02:52:54 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host- ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7fbd8405db18> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) RuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2561 (Sat Sep 15 02:53:03 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2561 (Sat Sep 15 02:53:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2e14c6de", "local_conf_timestamp": 2561, "host-ts": 2561}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2552 (Sat Sep 15 02:52:54 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2552 (Sat Sep 15 02:52:54 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host- --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 53, in restart_he_vm _wait_for_engine_health(host) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 185, in _wait_for_engine_health testlib.assert_true_within_long(lambda: any( File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) 'could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2561 (Sat Sep 15 02:53:03 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2561 (Sat Sep 15 02:53:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2e14c6de", "local_conf_timestamp": 2561, "host-ts": 2561}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2552 (Sat Sep 15 02:52:54 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2552 (Sat Sep 15 02:52:54 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:ed47f4dc-58b7-4cd6-8718-820f742fbaa8:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:ed47f4dc-58b7-4cd6-8718-820f742fbaa8:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 9face572 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9face572 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 9face572 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2397 (Sat Sep 15 02:50:19 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2397 (Sat Sep 15 02:50:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5f097b86", "local_conf_timestamp": 2397, "host-ts": 2397}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2392 (Sat Sep 15 02:50:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2392 (Sat Sep 15 02:50:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "524cc788", "local_conf_timestamp": 2392, "host-ts": 2392}, "global_maintenance": true}\n\nroot: INFO: * Shutting down HE VM on host: lago-he-basic-ansible-suite-master-host-0\nlago.ssh: DEBUG: start task:9d2bbc93-b674-4f8f-9cff-0ed017480579:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:9d2bbc93-b674-4f8f-9cff-0ed017480579:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running a03b5ad2 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-shutdown\nlago.ssh: DEBUG: Command a03b5ad2 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be down...\nlago.ssh: DEBUG: start task:c6f14484-bc15-46ae-a11a-d71e7d7736e6:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:c6f14484-bc15-46ae-a11a-d71e7d7736e6:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running a0d7f720 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a0d7f720 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a0d7f720 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2397 (Sat Sep 15 02:50:19 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2397 (Sat Sep 15 02:50:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5f097b86", "local_conf_timestamp": 2397, "host-ts": 2397}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2392 (Sat Sep 15 02:50:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2392 (Sat Sep 15 02:50:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "524cc788", "local_conf_timestamp": 2392, "host-ts": 2392}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:b7f4b056-1ffd-4395-86d7-cbdba57b299a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:b7f4b056-1ffd-4395-86d7-cbdba57b299a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running a76cfd56 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a76cfd56 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a76cfd56 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2407 (Sat Sep 15 02:50:29 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2407 (Sat Sep 15 02:50:29 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f828e9c5", "local_conf_timestamp": 2407, "host-ts": 2407}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2402 (Sat Sep 15 02:50:24 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2402 (Sat Sep 15 02:50:25 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d36639d7", "local_conf_timestamp": 2402, "host-ts": 2402}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:0be59cd9-e4cb-4839-9891-894d2a5f611e:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:0be59cd9-e4cb-4839-9891-894d2a5f611e:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running a9bc073c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a9bc073c on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a9bc073c on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2417 (Sat Sep 15 02:50:39 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2417 (Sat Sep 15 02:50:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "42b65cd2", "local_conf_timestamp": 2417, "host-ts": 2417}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2412 (Sat Sep 15 02:50:34 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2412 (Sat Sep 15 02:50:34 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "db6596aa", "local_conf_timestamp": 2412, "host-ts": 2412}, "global_maintenance": true}\n\nroot: INFO: * VM is down.\nroot: INFO: * Stopping services...\nlago.ssh: DEBUG: start task:a7f11961-66fe-4a3e-8b5c-03640709deb3:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:a7f11961-66fe-4a3e-8b5c-03640709deb3:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running aa528b4e on lago-he-basic-ansible-suite-master-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command aa528b4e on lago-he-basic-ansible-suite-master-host-0 returned with 0\nroot: INFO: * Starting services...\nlago.ssh: DEBUG: start task:3470e088-3896-444c-9480-28f84251a111:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:3470e088-3896-444c-9480-28f84251a111:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running b42c375a on lago-he-basic-ansible-suite-master-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command b42c375a on lago-he-basic-ansible-suite-master-host-0 returned with 0\nroot: INFO: * Waiting for agent to be ready...\nlago.ssh: DEBUG: start task:c551d6c2-0baf-48c5-b907-f31f4391d1c6:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:c551d6c2-0baf-48c5-b907-f31f4391d1c6:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running b677b14c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command b677b14c on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command b677b14c on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:5dfa64c5-57dd-43f8-b09b-5e6549669059:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:5dfa64c5-57dd-43f8-b09b-5e6549669059:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running bcc603e6 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command bcc603e6 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command bcc603e6 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:4100a51d-ba55-4808-ac40-f68b20c91e05:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:4100a51d-ba55-4808-ac40-f68b20c91e05:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running beff4df2 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command beff4df2 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command beff4df2 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:46fa8f10-bbc9-433e-9b53-0029800375c7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:46fa8f10-bbc9-433e-9b53-0029800375c7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running c127b556 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command c127b556 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command c127b556 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:d5c1c51b-f487-4b59-8079-bdb59bdfcfa7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:d5c1c51b-f487-4b59-8079-bdb59bdfcfa7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running c360ebd0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command c360ebd0 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command c360ebd0 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:2dbd6214-28f6-4736-9a97-0e98076457c1:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:2dbd6214-28f6-4736-9a97-0e98076457c1:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running c58e6b3a on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command c58e6b3a on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command c58e6b3a on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:502b040a-af3b-4f44-aed9-9dbce56933d2:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:502b040a-af3b-4f44-aed9-9dbce56933d2:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running c7b6c876 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command c7b6c876 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command c7b6c876 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:51e6fde8-4042-4144-930e-252299ee405c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:51e6fde8-4042-4144-930e-252299ee405c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running c9e638c0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command c9e638c0 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command c9e638c0 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:508cff8b-8c2a-4584-b493-d2679f10ad82:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:508cff8b-8c2a-4584-b493-d2679f10ad82:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running cc0c3816 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command cc0c3816 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command cc0c3816 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:6033b3fb-f21b-447d-a137-04e7784aab4a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:6033b3fb-f21b-447d-a137-04e7784aab4a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running ce4de886 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command ce4de886 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ce4de886 on lago-he-basic-ansible-suite-master-host-0 output:\n \n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\n\n--== Host lago-he-basic-ansible-suite-master-host-0 (id: 1) status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : True\nHostname : lago-he-basic-ansible-suite-master-host-0\nHost ID : 1\nEngine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}\nScore : 0\nstopped : False\nLocal maintenance : False\ncrc32 : 1bfd8aa6\nlocal_conf_timestamp : 2481\nHost timestamp : 2481\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2481 (Sat Sep 15 02:51:43 2018)\n\thost-id=1\n\tscore=0\n\tvm_conf_refresh_time=2481 (Sat Sep 15 02:51:43 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=ReinitializeFSM\n\tstopped=False\n\n\n--== Host lago-he-basic-ansible-suite-master-host-1 (id: 2) status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : False\nHostname : lago-he-basic-ansible-suite-master-host-1\nHost ID : 2\nEngine status : unknown stale-data\nScore : 3400\nstopped : False\nLocal maintenance : False\ncrc32 : eb6c75a4\nlocal_conf_timestamp : 2472\nHost timestamp : 2472\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2472 (Sat Sep 15 02:51:34 2018)\n\thost-id=2\n\tscore=3400\n\tvm_conf_refresh_time=2472 (Sat Sep 15 02:51:34 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=GlobalMaintenance\n\tstopped=False\n\n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\nroot: INFO: * Agent is ready.\nroot: INFO: * Starting VM...\nlago.ssh: DEBUG: start task:4b642e15-e2d0-441d-b6c7-6726e4671726:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:4b642e15-e2d0-441d-b6c7-6726e4671726:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running cecfa204 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-start\nlago.ssh: DEBUG: Command cecfa204 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command cecfa204 on lago-he-basic-ansible-suite-master-host-0 output:\n VM exists and is down, cleaning up and restarting\n\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be UP...\nlago.ssh: DEBUG: start task:46acaf72-3d13-4871-abd2-af9b3e9be8e0:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:46acaf72-3d13-4871-abd2-af9b3e9be8e0:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running d022e580 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command d022e580 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command d022e580 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2481 (Sat Sep 15 02:51:43 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=2481 (Sat Sep 15 02:51:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=ReinitializeFSM\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "1bfd8aa6", "local_conf_timestamp": 2481, "host-ts": 2481}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2472 (Sat Sep 15 02:51:34 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2472 (Sat Sep 15 02:51:34 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eb6c75a4", "local_conf_timestamp": 2472, "host-ts": 2472}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:9aacbec2-ebe2-4adb-980c-28909ccec04a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:9aacbec2-ebe2-4adb-980c-28909ccec04a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running d698d1ea on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command d698d1ea on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command d698d1ea on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2491 (Sat Sep 15 02:51:53 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2491 (Sat Sep 15 02:51:53 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e4fd4d62", "local_conf_timestamp": 2491, "host-ts": 2491}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2482 (Sat Sep 15 02:51:44 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2482 (Sat Sep 15 02:51:45 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "937b423f", "local_conf_timestamp": 2482, "host-ts": 2482}, "global_maintenance": true}\n\nroot: INFO: * VM is UP.\nroot: INFO: * Waiting for engine to start...\nlago.ssh: DEBUG: start task:a1f19e92-cf1d-461b-8dfe-54f39840068c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:a1f19e92-cf1d-461b-8dfe-54f39840068c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running d7183cd2 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command d7183cd2 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command d7183cd2 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2491 (Sat Sep 15 02:51:53 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2491 (Sat Sep 15 02:51:53 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e4fd4d62", "local_conf_timestamp": 2491, "host-ts": 2491}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2482 (Sat Sep 15 02:51:44 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2482 (Sat Sep 15 02:51:45 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "937b423f", "local_conf_timestamp": 2482, "host-ts": 2482}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:7b048778-3a8f-468f-a3ba-40adba395ce2:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:7b048778-3a8f-468f-a3ba-40adba395ce2:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running dd98827e on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command dd98827e on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command dd98827e on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2501 (Sat Sep 15 02:52:03 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2502 (Sat Sep 15 02:52:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd877c52", "local_conf_timestamp": 2502, "host-ts": 2501}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2492 (Sat Sep 15 02:51:55 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2493 (Sat Sep 15 02:51:55 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ec7fddd4", "local_conf_timestamp": 2493, "host-ts": 2492}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:dec50117-7911-436c-92ab-23a6e01b9797:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:dec50117-7911-436c-92ab-23a6e01b9797:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running dfee1836 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command dfee1836 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command dfee1836 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2511 (Sat Sep 15 02:52:13 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2511 (Sat Sep 15 02:52:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c163b966", "local_conf_timestamp": 2511, "host-ts": 2511}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2492 (Sat Sep 15 02:51:55 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2493 (Sat Sep 15 02:51:55 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ec7fddd4", "local_conf_timestamp": 2493, "host-ts": 2492}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:3d54d54a-3318-45ce-9635-fa7c36c532e9:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:3d54d54a-3318-45ce-9635-fa7c36c532e9:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running e24c7e9c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e24c7e9c on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command e24c7e9c on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2511 (Sat Sep 15 02:52:13 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2511 (Sat Sep 15 02:52:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c163b966", "local_conf_timestamp": 2511, "host-ts": 2511}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2502 (Sat Sep 15 02:52:04 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2502 (Sat Sep 15 02:52:04 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fdda37f9", "local_conf_timestamp": 2502, "host-ts": 2502}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:081e05c7-1491-4111-a828-dcdfb7750adb:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:081e05c7-1491-4111-a828-dcdfb7750adb:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running e4a9c406 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e4a9c406 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command e4a9c406 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2511 (Sat Sep 15 02:52:13 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2511 (Sat Sep 15 02:52:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c163b966", "local_conf_timestamp": 2511, "host-ts": 2511}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2502 (Sat Sep 15 02:52:04 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2502 (Sat Sep 15 02:52:04 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fdda37f9", "local_conf_timestamp": 2502, "host-ts": 2502}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ed1622e6-c317-4d65-878b-c034ae4c684a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:ed1622e6-c317-4d65-878b-c034ae4c684a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running e72300d0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e72300d0 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command e72300d0 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2521 (Sat Sep 15 02:52:23 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2521 (Sat Sep 15 02:52:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "77dc31e3", "local_conf_timestamp": 2521, "host-ts": 2521}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2512 (Sat Sep 15 02:52:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2512 (Sat Sep 15 02:52:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f5d99884", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:b0e31724-b7d5-4e10-ba19-b85cd6a18237:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:b0e31724-b7d5-4e10-ba19-b85cd6a18237:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running e9946412 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e9946412 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command e9946412 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2521 (Sat Sep 15 02:52:23 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2521 (Sat Sep 15 02:52:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "77dc31e3", "local_conf_timestamp": 2521, "host-ts": 2521}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2512 (Sat Sep 15 02:52:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2512 (Sat Sep 15 02:52:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f5d99884", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:a3e7508c-a4ab-4711-94c6-9274d4dd866a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:a3e7508c-a4ab-4711-94c6-9274d4dd866a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running ebdc77f0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ebdc77f0 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ebdc77f0 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2521 (Sat Sep 15 02:52:23 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2521 (Sat Sep 15 02:52:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "77dc31e3", "local_conf_timestamp": 2521, "host-ts": 2521}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2512 (Sat Sep 15 02:52:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2512 (Sat Sep 15 02:52:14 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f5d99884", "local_conf_timestamp": 2512, "host-ts": 2512}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:07a8ca1d-6dae-49f5-9a41-8ff5113c3330:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:07a8ca1d-6dae-49f5-9a41-8ff5113c3330:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running ee2ea7e4 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ee2ea7e4 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ee2ea7e4 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2531 (Sat Sep 15 02:52:33 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2531 (Sat Sep 15 02:52:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "65a02926", "local_conf_timestamp": 2531, "host-ts": 2531}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2522 (Sat Sep 15 02:52:24 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2522 (Sat Sep 15 02:52:24 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eddd6903", "local_conf_timestamp": 2522, "host-ts": 2522}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ff262ca1-ab96-4d86-b032-6684126b2551:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:ff262ca1-ab96-4d86-b032-6684126b2551:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running f06d4998 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f06d4998 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command f06d4998 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2531 (Sat Sep 15 02:52:33 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2531 (Sat Sep 15 02:52:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "65a02926", "local_conf_timestamp": 2531, "host-ts": 2531}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2522 (Sat Sep 15 02:52:24 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2522 (Sat Sep 15 02:52:24 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eddd6903", "local_conf_timestamp": 2522, "host-ts": 2522}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:0bd6fcfe-a362-4cbb-9030-45e36f7121a3:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:0bd6fcfe-a362-4cbb-9030-45e36f7121a3:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running f2d07494 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f2d07494 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command f2d07494 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2541 (Sat Sep 15 02:52:43 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2541 (Sat Sep 15 02:52:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ad4637d", "local_conf_timestamp": 2541, "host-ts": 2541}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2532 (Sat Sep 15 02:52:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2533 (Sat Sep 15 02:52:35 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "92d9f6e8", "local_conf_timestamp": 2533, "host-ts": 2532}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:7221ff8d-1909-4fcf-87c8-ef62ef65ed2c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:7221ff8d-1909-4fcf-87c8-ef62ef65ed2c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running f544a65a on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f544a65a on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command f544a65a on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2541 (Sat Sep 15 02:52:43 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2541 (Sat Sep 15 02:52:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ad4637d", "local_conf_timestamp": 2541, "host-ts": 2541}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2532 (Sat Sep 15 02:52:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2533 (Sat Sep 15 02:52:35 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "92d9f6e8", "local_conf_timestamp": 2533, "host-ts": 2532}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:28d5ce55-c074-4848-aef7-6a9543351375:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:28d5ce55-c074-4848-aef7-6a9543351375:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running f7b68962 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f7b68962 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command f7b68962 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2541 (Sat Sep 15 02:52:43 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2541 (Sat Sep 15 02:52:43 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "1ad4637d", "local_conf_timestamp": 2541, "host-ts": 2541}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2532 (Sat Sep 15 02:52:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2533 (Sat Sep 15 02:52:35 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "92d9f6e8", "local_conf_timestamp": 2533, "host-ts": 2532}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:70cda777-6455-4ebe-ad9e-c50880c6f5ce:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:70cda777-6455-4ebe-ad9e-c50880c6f5ce:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running fa0f7b06 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fa0f7b06 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command fa0f7b06 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2551 (Sat Sep 15 02:52:53 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2551 (Sat Sep 15 02:52:53 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1890ef91", "local_conf_timestamp": 2551, "host-ts": 2551}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2542 (Sat Sep 15 02:52:44 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2542 (Sat Sep 15 02:52:44 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ddd48a0d", "local_conf_timestamp": 2542, "host-ts": 2542}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:c974937c-559f-41b0-b28b-dd0315b74d79:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:c974937c-559f-41b0-b28b-dd0315b74d79:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running fc7520bc on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fc7520bc on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command fc7520bc on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2551 (Sat Sep 15 02:52:53 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2551 (Sat Sep 15 02:52:53 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1890ef91", "local_conf_timestamp": 2551, "host-ts": 2551}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2542 (Sat Sep 15 02:52:44 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2542 (Sat Sep 15 02:52:44 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ddd48a0d", "local_conf_timestamp": 2542, "host-ts": 2542}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ccf674f8-d12b-4176-a81e-8fa962e61ea7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:ccf674f8-d12b-4176-a81e-8fa962e61ea7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running fed087e8 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fed087e8 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command fed087e8 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2561 (Sat Sep 15 02:53:03 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2561 (Sat Sep 15 02:53:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2e14c6de", "local_conf_timestamp": 2561, "host-ts": 2561}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2552 (Sat Sep 15 02:52:54 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2552 (Sat Sep 15 02:52:54 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7fbd8405db18>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda>\n for k, v in _get_he_status(host).items()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status\n raise RuntimeError(\'could not parse JSON: %s\' % ret.out)\nRuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2561 (Sat Sep 15 02:53:03 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2561 (Sat Sep 15 02:53:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "2e14c6de", "local_conf_timestamp": 2561, "host-ts": 2561}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2552 (Sat Sep 15 02:52:54 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2552 (Sat Sep 15 02:52:54 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-\n--------------------- >> end captured logging << ---------------------'

1 1

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #1946
by jenkins＠jenkins.phx.ovirt.org 15 Sep '18

15 Sep '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 9c187d6e9145db35423f495b553304ceb75489cb (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 9c187d6e9145db35423f495b553304ceb75489cb Commit message: "Add jobs for ovs-cni" > git rev-list --no-walk 9c187d6e9145db35423f495b553304ceb75489cb # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins4484725594797596479.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7/packages error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7/packages' Build step 'Execute shell' marked build as failure

1 1

[CQ]: 94317,2 (vdsm) failed "ovirt-master" system tests
by oVirt Jenkins 15 Sep '18

15 Sep '18

Change 94317,2 (vdsm) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/94317/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10323/

1 0

oVirt infra daily report - unstable production jobs - 583
by jenkins＠jenkins.phx.ovirt.org 15 Sep '18

15 Sep '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/583//artifact/exported-a… Cheers, Jenkins

1 0

[CQ]: 94307, 6 (ovirt-provider-ovn) failed "ovirt-master" system tests
by oVirt Jenkins 14 Sep '18

14 Sep '18

Change 94307,6 (ovirt-provider-ovn) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/94307/6 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10318/

1 0

[oVirt Jenkins] ovirt-system-tests_he-basic-ansible-suite-4.2 - Build # 565 - Failure!
by jenkins＠jenkins.phx.ovirt.org 14 Sep '18

14 Sep '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4.2/… Build Number: 565 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #565 [Milan Zamazal] Update test-vm.ovf file ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 010_local_maintenance_cli.local_maintenance Error Message: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... lago.ssh: DEBUG: start task:0874be8a-4d1f-4242-8c58-343d3bf8718a:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:0874be8a-4d1f-4242-8c58-343d3bf8718a:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 2d38dc7e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 2d38dc7e on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 2d38dc7e on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3333 (Fri Sep 14 04:21:19 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=3333 (Fri Sep 14 04:21:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b74983cd", "local_conf_timestamp": 3333, "host-ts": 3333}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3331 (Fri Sep 14 04:21:18 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3331 (Fri Sep 14 04:21:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "547f6cbb", "local_conf_timestamp": 3331, "host-ts": 3331}, "global_maintenance": false} lago.ssh: DEBUG: start task:de4315ba-d73d-4a51-8315-8f960c2a0e35:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:de4315ba-d73d-4a51-8315-8f960c2a0e35:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 2db65230 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local lago.ssh: DEBUG: Command 2db65230 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 root: INFO: * Waiting for engine to migrate... lago.ssh: DEBUG: start task:f48ea071-64ea-48d4-b8a4-91c01f7d0bca:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:f48ea071-64ea-48d4-b8a4-91c01f7d0bca:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 2e3e3952 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 2e3e3952 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 2e3e3952 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3343 (Fri Sep 14 04:21:29 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3343 (Fri Sep 14 04:21:29 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "490d51f6", "local_conf_timestamp": 3343, "host-ts": 3343}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3341 (Fri Sep 14 04:21:27 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3341 (Fri Sep 14 04:21:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fae41c6b", "local_conf_timestamp": 3341, "host-ts": 3341}, "global_maintenance": false} lago.ssh: DEBUG: start task:a395325a-1bb1-4628-afe9-3c044c03a15f:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:a395325a-1bb1-4628-afe9-3c044c03a15f:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 34d2d908 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 34d2d908 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 34d2d908 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3353 (Fri Sep 14 04:21:39 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3353 (Fri Sep 14 04:21:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7eb24832", "local_conf_timestamp": 3353, "host-ts": 3353}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3351 (Fri Sep 14 04:21:37 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3351 (Fri Sep 14 04:21:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c78129c7", "local_conf_timestamp": 3351, "host-ts": 3351}, "global_maintenance": false} lago.ssh: DEBUG: start task:bcb68d9f-1dd3-4a4a-bc77-4dd79e76bbed:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:bcb68d9f-1dd3-4a4a-bc77-4dd79e76bbed:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 372bbbf2 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 372bbbf2 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 372bbbf2 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3353 (Fri Sep 14 04:21:39 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3353 (Fri Sep 14 04:21:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7eb24832", "local_conf_timestamp": 3353, "host-ts": 3353}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3351 (Fri Sep 14 04:21:37 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3351 (Fri Sep 14 04:21:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c78129c7", "local_conf_timestamp": 3351, "host-ts": 3351}, "global_maintenance": false} lago.ssh: DEBUG: start task:09f83e32-8ee3-4d44-9da3-4223626e2ed9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:09f83e32-8ee3-4d44-9da3-4223626e2ed9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 39875618 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 39875618 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 39875618 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3353 (Fri Sep 14 04:21:39 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3353 (Fri Sep 14 04:21:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7eb24832", "local_conf_timestamp": 3353, "host-ts": 3353}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3351 (Fri Sep 14 04:21:37 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3351 (Fri Sep 14 04:21:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c78129c7", "local_conf_timestamp": 3351, "host-ts": 3351}, "global_maintenance": false} lago.ssh: DEBUG: start task:cc21a5f2-c7d4-4df7-9212-efd5f0ddfb67:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:cc21a5f2-c7d4-4df7-9212-efd5f0ddfb67:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 3c072cb0 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 3c072cb0 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 3c072cb0 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3363 (Fri Sep 14 04:21:49 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3364 (Fri Sep 14 04:21:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d0ae2b36", "local_conf_timestamp": 3364, "host-ts": 3363}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3361 (Fri Sep 14 04:21:47 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3361 (Fri Sep 14 04:21:48 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "064376e1", "local_conf_timestamp": 3361, "host-ts": 3361}, "global_maintenance": false} lago.ssh: DEBUG: start task:ed320820-2eb4-4618-ab75-f0607ad763c9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:ed320820-2eb4-4618-ab75-f0607ad763c9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 3e643520 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 3e643520 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 3e643520 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3363 (Fri Sep 14 04:21:49 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3364 (Fri Sep 14 04:21:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d0ae2b36", "local_conf_timestamp": 3364, "host-ts": 3363}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3361 (Fri Sep 14 04:21:47 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3361 (Fri Sep 14 04:21:48 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "064376e1", "local_conf_timestamp": 3361, "host-ts": 3361}, "global_maintenance": false} lago.ssh: DEBUG: start task:5de6d339-db62-4214-8c40-42134ea48ff6:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:5de6d339-db62-4214-8c40-42134ea48ff6:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 40baf412 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 40baf412 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 40baf412 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1c53d696", "local_conf_timestamp": 3372, "host-ts": 3371}, "global_maintenance": false} lago.ssh: DEBUG: start task:51240133-3070-4a87-9f8b-b059cf3f9f0b:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:51240133-3070-4a87-9f8b-b059cf3f9f0b:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 43075d14 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 43075d14 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 43075d14 on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1c53d696", "local_conf_timestamp": 3372, "host-ts": 3371}, "global_maintenance": false} lago.ssh: DEBUG: start task:e27a9a75-6f74-4c35-b53b-64e3224449b0:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: end task:e27a9a75-6f74-4c35-b53b-64e3224449b0:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0: lago.ssh: DEBUG: Running 4564db7c on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 4564db7c on lago-he-basic-ansible-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 4564db7c on lago-he-basic-ansible-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7fc20ea63a28> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 36, in <lambda> testlib.assert_true_within_long(lambda: _get_he_status(host) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 98, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) RuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 125, in local_maintenance _wait_for_engine_migration(host, he_index, "bad", "Migration Destination") File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 36, in _wait_for_engine_migration testlib.assert_true_within_long(lambda: _get_he_status(host) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 36, in <lambda> testlib.assert_true_within_long(lambda: _get_he_status(host) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 98, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) 'could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nlago.ssh: DEBUG: start task:0874be8a-4d1f-4242-8c58-343d3bf8718a:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:0874be8a-4d1f-4242-8c58-343d3bf8718a:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 2d38dc7e on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 2d38dc7e on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 2d38dc7e on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3333 (Fri Sep 14 04:21:19 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=3333 (Fri Sep 14 04:21:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b74983cd", "local_conf_timestamp": 3333, "host-ts": 3333}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3331 (Fri Sep 14 04:21:18 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3331 (Fri Sep 14 04:21:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "547f6cbb", "local_conf_timestamp": 3331, "host-ts": 3331}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:de4315ba-d73d-4a51-8315-8f960c2a0e35:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:de4315ba-d73d-4a51-8315-8f960c2a0e35:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 2db65230 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local\nlago.ssh: DEBUG: Command 2db65230 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nroot: INFO: * Waiting for engine to migrate...\nlago.ssh: DEBUG: start task:f48ea071-64ea-48d4-b8a4-91c01f7d0bca:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:f48ea071-64ea-48d4-b8a4-91c01f7d0bca:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 2e3e3952 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 2e3e3952 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 2e3e3952 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3343 (Fri Sep 14 04:21:29 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3343 (Fri Sep 14 04:21:29 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "490d51f6", "local_conf_timestamp": 3343, "host-ts": 3343}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3341 (Fri Sep 14 04:21:27 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3341 (Fri Sep 14 04:21:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fae41c6b", "local_conf_timestamp": 3341, "host-ts": 3341}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:a395325a-1bb1-4628-afe9-3c044c03a15f:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:a395325a-1bb1-4628-afe9-3c044c03a15f:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 34d2d908 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 34d2d908 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 34d2d908 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3353 (Fri Sep 14 04:21:39 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3353 (Fri Sep 14 04:21:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7eb24832", "local_conf_timestamp": 3353, "host-ts": 3353}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3351 (Fri Sep 14 04:21:37 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3351 (Fri Sep 14 04:21:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c78129c7", "local_conf_timestamp": 3351, "host-ts": 3351}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:bcb68d9f-1dd3-4a4a-bc77-4dd79e76bbed:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:bcb68d9f-1dd3-4a4a-bc77-4dd79e76bbed:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 372bbbf2 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 372bbbf2 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 372bbbf2 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3353 (Fri Sep 14 04:21:39 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3353 (Fri Sep 14 04:21:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7eb24832", "local_conf_timestamp": 3353, "host-ts": 3353}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3351 (Fri Sep 14 04:21:37 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3351 (Fri Sep 14 04:21:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c78129c7", "local_conf_timestamp": 3351, "host-ts": 3351}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:09f83e32-8ee3-4d44-9da3-4223626e2ed9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:09f83e32-8ee3-4d44-9da3-4223626e2ed9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 39875618 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 39875618 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 39875618 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3353 (Fri Sep 14 04:21:39 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3353 (Fri Sep 14 04:21:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "7eb24832", "local_conf_timestamp": 3353, "host-ts": 3353}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3351 (Fri Sep 14 04:21:37 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3351 (Fri Sep 14 04:21:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c78129c7", "local_conf_timestamp": 3351, "host-ts": 3351}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:cc21a5f2-c7d4-4df7-9212-efd5f0ddfb67:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:cc21a5f2-c7d4-4df7-9212-efd5f0ddfb67:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 3c072cb0 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 3c072cb0 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 3c072cb0 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3363 (Fri Sep 14 04:21:49 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3364 (Fri Sep 14 04:21:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d0ae2b36", "local_conf_timestamp": 3364, "host-ts": 3363}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3361 (Fri Sep 14 04:21:47 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3361 (Fri Sep 14 04:21:48 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "064376e1", "local_conf_timestamp": 3361, "host-ts": 3361}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:ed320820-2eb4-4618-ab75-f0607ad763c9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:ed320820-2eb4-4618-ab75-f0607ad763c9:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 3e643520 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 3e643520 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 3e643520 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3363 (Fri Sep 14 04:21:49 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3364 (Fri Sep 14 04:21:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "d0ae2b36", "local_conf_timestamp": 3364, "host-ts": 3363}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3361 (Fri Sep 14 04:21:47 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3361 (Fri Sep 14 04:21:48 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "064376e1", "local_conf_timestamp": 3361, "host-ts": 3361}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:5de6d339-db62-4214-8c40-42134ea48ff6:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:5de6d339-db62-4214-8c40-42134ea48ff6:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 40baf412 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 40baf412 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 40baf412 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1c53d696", "local_conf_timestamp": 3372, "host-ts": 3371}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:51240133-3070-4a87-9f8b-b059cf3f9f0b:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:51240133-3070-4a87-9f8b-b059cf3f9f0b:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 43075d14 on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 43075d14 on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 43075d14 on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1c53d696", "local_conf_timestamp": 3372, "host-ts": 3371}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:e27a9a75-6f74-4c35-b53b-64e3224449b0:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:e27a9a75-6f74-4c35-b53b-64e3224449b0:Get ssh client for lago-he-basic-ansible-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 4564db7c on lago-he-basic-ansible-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 4564db7c on lago-he-basic-ansible-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 4564db7c on lago-he-basic-ansible-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7fc20ea63a28>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 36, in <lambda>\n testlib.assert_true_within_long(lambda: _get_he_status(host)\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 98, in _get_he_status\n raise RuntimeError(\'could not parse JSON: %s\' % ret.out)\nRuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3373 (Fri Sep 14 04:21:59 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3373 (Fri Sep 14 04:21:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e8321137", "local_conf_timestamp": 3373, "host-ts": 3373}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3371 (Fri Sep 14 04:21:58 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=3372 (Fri Sep 14 04:21:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason\n--------------------- >> end captured logging << ---------------------'

1 1

[CQ]: 94344, 1 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 14 Sep '18

14 Sep '18

A system test invoked by the "ovirt-master" change queue including change 94344,1 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 94081,3 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 94081,3 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/94344/1 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/94081/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10314/

1 0

[JIRA] (OVIRT-2498) Failing KubeVirt CI
by Petr Kotas (oVirt JIRA) 14 Sep '18

14 Sep '18

Petr Kotas created OVIRT-2498: --------------------------------- Summary: Failing KubeVirt CI Key: OVIRT-2498 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2498 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Petr Kotas Assignee: infra Hi, I am working on fixing the issues on the KubeVirt e2e test suites. This task is directly related to unstable CI, due to unknown errors. The progress is reported in the CNV trello: https://trello.com/c/HNXcMEQu/161-epic-improve-ci I am creating this issue since the KubeVirt experience random timeouts on random tests most of the times when test suites run. The issue from outside is showing as timeouts on difference part of tests. Sometimes the tests fails in set up phase, again due to random timeout. The example in the link bellow timed out for network connection on localhost. [check-patch.k8s-1.11.0-dev.el7.x86_64] requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60) Example of failing test suites is here https://jenkins.ovirt.org/job/kubevirt_kubevirt_standard-check-pr/1916/cons… The list of errors related to the failing CI can be found in my notes https://docs.google.com/document/d/1_ll1DOMHgCRHn_Df9i4uvtRFyMK-bDCHEeGfJFT… I am not sure whether KubeVirt already shared the resource requirements, so I provide short summary: *Resources for KubeVirt e2e tests:* - at least 12GB of RAM - we start 3 nodes (3 docker images) each require 4GB of RAM - exposed /dev/kvm to enable native virtualization - cached images, since these are used to build the test cluster: - kubevirtci/os-3.10.0-crio:latest - kubevirtci/os-3.10.0-multus:latest - kubevirtci/os-3.10.0:latest - kubevirtci/k8s-1.10.4:latest - kubevirtci/k8s-multus-1.11.1:latest - kubevirtci/k8s-1.11.0:latest How can we overcome this? Can we work together to build a suitable requirements for running the tests so it passes each time? Kind regards, Petr Kotas -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[CQ]: 94317, 2 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 14 Sep '18

14 Sep '18

A system test invoked by the "ovirt-master" change queue including change 94317,2 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 94249,9 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 94249,9 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/94317/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/94249/9 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10311/

2 1

[CQ]: 5650929 (ovirt-ansible-v2v-conversion-host) failed "ovirt-4.2" system tests
by oVirt Jenkins 14 Sep '18

14 Sep '18

Change 5650929 (ovirt-ansible-v2v-conversion-host) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://github.com/oVirt/ovirt-ansible-v2v-conversion-host/commit/565092928… For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3112/

2 1

[JIRA] (OVIRT-2497) ovirt-master in project vdsm failing on migrate vm test
by Dafna Ron (oVirt JIRA) 14 Sep '18

14 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2497?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2497: ---------------------------------- the patch that is root cause may actually be the cause because its part of the change. but it can also be the api patch. I added a comment in the mail I sent to dev but forgot to update the ticket > ovirt-master in project vdsm failing on migrate vm test > ------------------------------------------------------- > > Key: OVIRT-2497 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2497 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_code_regression, ost_failures > > we have a failure in migrate vm for project vdsm > full logs can be found here: > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10309/artifa… > reported patch and root cause patch do not seem related to the failure. > I suspect this is the change that caused the issue: > https://gerrit.ovirt.org/#/c/94248/ > error: https://pastebin.com/XNGFgfvL -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2497) ovirt-master in project vdsm failing on migrate vm test
by Barak Korren (oVirt JIRA) 14 Sep '18

14 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2497?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-2497: ------------------------------------- בתאריך יום ו׳, 14 בספט׳ 2018, 15:20, מאת Dafna Ron (oVirt JIRA) ‏< So why did the root cause patch fail? I suspect this is the change that caused the issue: > ovirt-master in project vdsm failing on migrate vm test > ------------------------------------------------------- > > Key: OVIRT-2497 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2497 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_code_regression, ost_failures > > we have a failure in migrate vm for project vdsm > full logs can be found here: > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10309/artifa… > reported patch and root cause patch do not seem related to the failure. > I suspect this is the change that caused the issue: > https://gerrit.ovirt.org/#/c/94248/ > error: https://pastebin.com/XNGFgfvL -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2497) ovirt-master in project vdsm failing on migrate vm test
by Dafna Ron (oVirt JIRA) 14 Sep '18

14 Sep '18

Dafna Ron created OVIRT-2497: -------------------------------- Summary: ovirt-master in project vdsm failing on migrate vm test Key: OVIRT-2497 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2497 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Dafna Ron Assignee: infra Priority: Highest we have a failure in migrate vm for project vdsm full logs can be found here: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10309/artifa… reported patch and root cause patch do not seem related to the failure. I suspect this is the change that caused the issue: https://gerrit.ovirt.org/#/c/94248/ error: https://pastebin.com/XNGFgfvL -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

2 1

[CQ]: 94333, 2 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 14 Sep '18

14 Sep '18

A system test invoked by the "ovirt-master" change queue including change 94333,2 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 94249,9 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 94249,9 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/94333/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/94249/9 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10309/

2 1

[JIRA] (OVIRT-2494) CQ Failure - Cannot add host
by Dafna Ron (oVirt JIRA) 14 Sep '18

14 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2494?page=com.atlassian.jira.… ] Dafna Ron updated OVIRT-2494: ----------------------------- Issue Type: Bug (was: Outage) > CQ Failure - Cannot add host > ---------------------------- > > Key: OVIRT-2494 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2494 > Project: oVirt - virtualization made easy > Issue Type: Bug > Components: Change Queue, OST > Reporter: Ehud Yonasi > Assignee: infra > Labels: cq-issue-traking, ost_failures > > Hi, > This is an example of a he basic suite master that fails on : > Error Message: > 00:26:36 Error while running thread > 00:26:36 Traceback (most recent call last): > 00:26:36 File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue > 00:26:36 queue.put({'return': func()}) > 00:26:36 File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-master/ovirt-system-tests/he-basic-suite-master/test-scenarios/002_bootstrap.py", line 379, in _add_he_host_4 > 00:26:36 deploy_hosted_engine=True, > 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 11530, in add > 00:26:36 return self._internal_add(host, headers, query, wait) > 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add > 00:26:36 return future.wait() if wait else future > 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait > 00:26:36 return self._code(response) > 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback > 00:26:36 self._check_fault(response) > 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault > 00:26:36 self._raise_error(response, body) > 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error > 00:26:36 raise error > 00:26:36 Error: Fault reason is "Operation Failed". Fault detail is "[Cannot add Host. You are using an unmanaged hosted engine VM. Please add the first storage domain in order to start the hosted engine import process.]". HTTP response code is 409. > Job url: > https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… > There is a patch which was merged: > https://gerrit.ovirt.org/#/c/94219/ > It did not pass CQ because of another failure which Ravi is working on ( Add cluster issue). -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[CQ]: 94337, 2 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 14 Sep '18

14 Sep '18

A system test invoked by the "ovirt-master" change queue including change 94337,2 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 94081,3 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 94081,3 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/94337/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/94081/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10307/

2 1

[JIRA] (OVIRT-2496) master ovirt-engine is failing on 002_bootstrap.add_hosts in upgrade suite
by Dafna Ron (oVirt JIRA) 14 Sep '18

14 Sep '18

Dafna Ron created OVIRT-2496: -------------------------------- Summary: master ovirt-engine is failing on 002_bootstrap.add_hosts in upgrade suite Key: OVIRT-2496 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2496 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Dafna Ron Assignee: infra Priority: Highest we have a new regression after the previous one was fixed. the issue is in the upgrade suite with test 002_bootstrap.add_hosts. full logs: https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_chang… error: https://pastebin.com/B5wxKTJy -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2495) vdsm-jsonrpc-java-client:jar:1.4.15 missing on artifactory
by Evgheni Dereveanchin (oVirt JIRA) 14 Sep '18

14 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2495?page=com.atlassian.jira.… ] Evgheni Dereveanchin reassigned OVIRT-2495: ------------------------------------------- Assignee: Evgheni Dereveanchin (was: infra) > vdsm-jsonrpc-java-client:jar:1.4.15 missing on artifactory > ---------------------------------------------------------- > > Key: OVIRT-2495 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2495 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Evgheni Dereveanchin > Assignee: Evgheni Dereveanchin > > The artifact is missing from our mirror for some reason as reported by [~pkliczew(a)redhat.com] and causing ovirt-engine CI to fail: > https://jenkins.ovirt.org/job/ovirt-engine_master_check-patch-fc28-x86_64/2… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2495) vdsm-jsonrpc-java-client:jar:1.4.15 missing on artifactory
by Evgheni Dereveanchin (oVirt JIRA) 14 Sep '18

14 Sep '18

Evgheni Dereveanchin created OVIRT-2495: ------------------------------------------- Summary: vdsm-jsonrpc-java-client:jar:1.4.15 missing on artifactory Key: OVIRT-2495 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2495 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Evgheni Dereveanchin Assignee: infra The artifact is missing from our mirror for some reason as reported by [~pkliczew(a)redhat.com] and causing ovirt-engine CI to fail: https://jenkins.ovirt.org/job/ovirt-engine_master_check-patch-fc28-x86_64/2… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[CQ]: 94332, 3 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 14 Sep '18

14 Sep '18

A system test invoked by the "ovirt-master" change queue including change 94332,3 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 94081,3 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 94081,3 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/94332/3 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/94081/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10305/

1 0

Build failed in Jenkins: system-sync_mirrors-epel-el6-x86_64 #1860
by jenkins＠jenkins.phx.ovirt.org 14 Sep '18

14 Sep '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/1860/displ…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 9c187d6e9145db35423f495b553304ceb75489cb (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 9c187d6e9145db35423f495b553304ceb75489cb Commit message: "Add jobs for ovs-cni" > git rev-list --no-walk 9c187d6e9145db35423f495b553304ceb75489cb # timeout=10 [system-sync_mirrors-epel-el6-x86_64] $ /bin/bash -xe /tmp/jenkins4338967625785528158.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror epel-el6 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7' Build step 'Execute shell' marked build as failure

1 1

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #1944
by jenkins＠jenkins.phx.ovirt.org 14 Sep '18

14 Sep '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 9c187d6e9145db35423f495b553304ceb75489cb (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 9c187d6e9145db35423f495b553304ceb75489cb Commit message: "Add jobs for ovs-cni" > git rev-list --no-walk 9c187d6e9145db35423f495b553304ceb75489cb # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins667231316923733405.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Resyncing repo: centos-updates-el7 Syncing yum repo centos-updates-el7 for arch: x86_64 Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el6/packages error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el6/packages' Build step 'Execute shell' marked build as failure

1 1

[CQ]: 94149, 4 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 14 Sep '18

14 Sep '18

A system test invoked by the "ovirt-master" change queue including change 94149,4 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 94081,3 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 94081,3 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/94149/4 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/94081/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10301/

1 0

[CQ]: 94252, 7 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 14 Sep '18

14 Sep '18

A system test invoked by the "ovirt-master" change queue including change 94252,7 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 94249,9 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 94249,9 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/94252/7 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/94249/9 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10297/

1 0

[CQ]: 94249,9 (vdsm) failed "ovirt-master" system tests
by oVirt Jenkins 14 Sep '18

14 Sep '18

Change 94249,9 (vdsm) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/94249/9 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10292/

1 0

oVirt infra daily report - unstable production jobs - 582
by jenkins＠jenkins.phx.ovirt.org 14 Sep '18

14 Sep '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/582//artifact/exported-a… Cheers, Jenkins

1 0

[CQ]: 94288, 4 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 13 Sep '18

13 Sep '18

A system test invoked by the "ovirt-master" change queue including change 94288,4 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 94081,3 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 94081,3 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/94288/4 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/94081/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10288/

1 0

[CQ]: 94219, 21 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 13 Sep '18

13 Sep '18

A system test invoked by the "ovirt-master" change queue including change 94219,21 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 94081,3 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 94081,3 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/94219/21 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/94081/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10283/

1 0

[CQ]: 94306, 2 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 13 Sep '18

13 Sep '18

A system test invoked by the "ovirt-master" change queue including change 94306,2 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 94081,3 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 94081,3 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/94306/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/94081/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10278/

1 0

[JIRA] (OVIRT-2489) Add nmstate project to STDCI
by Ehud Yonasi (oVirt JIRA) 13 Sep '18

13 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2489?page=com.atlassian.jira.… ] Ehud Yonasi reassigned OVIRT-2489: ---------------------------------- Assignee: Ehud Yonasi (was: infra) > Add nmstate project to STDCI > ---------------------------- > > Key: OVIRT-2489 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2489 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Edward Haas > Assignee: Ehud Yonasi > > Hello, > nmstate is a new project that is aimed to replace most of the VDSM > networking implementation and hopefully used to manage k8s nodes networking. > It has raised from a joint effort between the NetworkManager team and oVirt > networking team. > We would like to utilize STDCI to run integration tests. > A PR has been opened for this: https://github.com/nmstate/nmstate/pull/76 > Could you please help us add it to the CI? > Thanks, > Edy. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2479) CQ Failure - Alot of ovirt-hosted-engine-setup messages.
by Ehud Yonasi (oVirt JIRA) 13 Sep '18

13 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2479?page=com.atlassian.jira.… ] Ehud Yonasi updated OVIRT-2479: ------------------------------- Assignee: Ehud Yonasi (was: infra) Resolution: Fixed Status: Done (was: To Do) > CQ Failure - Alot of ovirt-hosted-engine-setup messages. > -------------------------------------------------------- > > Key: OVIRT-2479 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2479 > Project: oVirt - virtualization made easy > Issue Type: Bug > Components: Change Queue, OST > Reporter: Ehud Yonasi > Assignee: Ehud Yonasi > Labels: cq-issue-traking, ost_failures > > Since 3rd september there were alot of changes coming out of the blue to CQ about ovirt-hosted-engine-setup. > while investigating why it happened, I found out that the remote repo was deleted somehow and was restored. > That restoration caused all the project patches all-at-once triggers hundreds of jobs in build-art + CQ. > In order to recover from that situation i had to find the last patch of that project and asked for a re-merge (ci re-merge please). > After that change and multiple runs of CQ it passed. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[CQ]: 93710, 5 (vdsm) failed "ovirt-4.2" system tests, but isn't the failure root cause
by oVirt Jenkins 13 Sep '18

13 Sep '18

A system test invoked by the "ovirt-4.2" change queue including change 93710,5 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 94186,2 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 94186,2 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/93710/5 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/94186/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3092/

2 1

[CQ]: 94294, 2 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 13 Sep '18

13 Sep '18

A system test invoked by the "ovirt-master" change queue including change 94294,2 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 94081,3 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 94081,3 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/94294/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/94081/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10274/

1 0

[JIRA] (OVIRT-2494) CQ Failure - Cannot add host
by Ehud Yonasi (oVirt JIRA) 13 Sep '18

13 Sep '18

Ehud Yonasi created OVIRT-2494: ---------------------------------- Summary: CQ Failure - Cannot add host Key: OVIRT-2494 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2494 Project: oVirt - virtualization made easy Issue Type: Outage Components: Change Queue, OST Reporter: Ehud Yonasi Assignee: infra Hi, This is an example of a he basic suite master that fails on : Error Message: 00:26:36 Error while running thread 00:26:36 Traceback (most recent call last): 00:26:36 File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue 00:26:36 queue.put({'return': func()}) 00:26:36 File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-master/ovirt-system-tests/he-basic-suite-master/test-scenarios/002_bootstrap.py", line 379, in _add_he_host_4 00:26:36 deploy_hosted_engine=True, 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 11530, in add 00:26:36 return self._internal_add(host, headers, query, wait) 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add 00:26:36 return future.wait() if wait else future 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait 00:26:36 return self._code(response) 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback 00:26:36 self._check_fault(response) 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault 00:26:36 self._raise_error(response, body) 00:26:36 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error 00:26:36 raise error 00:26:36 Error: Fault reason is "Operation Failed". Fault detail is "[Cannot add Host. You are using an unmanaged hosted engine VM. Please add the first storage domain in order to start the hosted engine import process.]". HTTP response code is 409. Job url: https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… There is a patch which was merged: https://gerrit.ovirt.org/#/c/94219/ It did not pass CQ because of another failure which Ravi is working on ( Add cluster issue). -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2492) ds-vdsm was not used for rhv CQ tests
by Evgheni Dereveanchin (oVirt JIRA) 13 Sep '18

13 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2492?page=com.atlassian.jira.… ] Evgheni Dereveanchin deleted OVIRT-2492: ---------------------------------------- > ds-vdsm was not used for rhv CQ tests > ------------------------------------- > > Key: OVIRT-2492 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2492 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2492) ds-vdsm was not used for rhv CQ tests
by Dafna Ron (oVirt JIRA) 13 Sep '18

13 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2492?page=com.atlassian.jira.… ] Dafna Ron updated OVIRT-2492: ----------------------------- Resolution: Fixed Status: Done (was: To Do) > ds-vdsm was not used for rhv CQ tests > ------------------------------------- > > Key: OVIRT-2492 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2492 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_failures, ost_packages > > we found that vdsm was downloaded from > http://download-node-02.eng.bos.redhat.com/rel-eng/repos/rhevm-4.2-rhel-7-p… > since its a ds project it should be going down from: > http://ci-web.eng.lab.tlv.redhat.com/repos/tested/rhv-4.2/rpm/el7/ > The reason it was downloaded is because package version numbers in official 4.2 is higher then the one in tested so reposync tool was taking it from the wrong repo. > I am adding patches to exclude official repo so that vdsm will download from tested. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[CQ]: 94159, 5 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 13 Sep '18

13 Sep '18

A system test invoked by the "ovirt-master" change queue including change 94159,5 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 94081,3 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 94081,3 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/94159/5 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/94081/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10271/

1 0

[ OST Failure Report ] [ oVirt Master (ovirt-engine) ] [ 11-09-2018 ] [ 002_bootstrap.add_cluster ]
by Dafna Ron 13 Sep '18

13 Sep '18

Hi, we have failures on both upgrade and basic suites for master. The patch reported as cause is: https://gerrit.ovirt.org/#/c/93345/10 - engine : Add finer grained monitoring thresholds for memory consumption on Hypervisors Ravi, can you please check this issue? You can see the logs here: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10187/artifa… Here is the error: 2018-09-11 02:08:44,511-04 WARN [org.ovirt.engine.core.bll.AddClusterCommand] (default task-2) [1362d97c-9d55-40d2-9b03-ebe93ae2fe67] Validation of action 'AddCluster' failed for user admin@internal-authz. Reasons: VAR__TYPE__CLUSTER,VAR __ACTION__CREATE,must be greater than or equal to 1,$groups [Ljava.lang.Class;@29a040fc,$message {javax.validation.constraints.Min.message},$payload [Ljava.lang.Class;@64094510,$value 1,ACTION_TYPE_FAILED_ATTRIBUTE_PATH,$path cluster.logM axMemoryUsedThreshold,$validatedValue 0 2018-09-11 02:08:44,511-04 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-2) [1362d97c-9d55-40d2-9b03-ebe93ae2fe67] method: runAction, params: [AddCluster, ManagementNetworkOnClusterOperationPara meters:{commandId='e5eb6cf6-4e3d-4f50-be6a-9fe18bfc4d97', user='null', commandType='Unknown'}], timeElapsed: 56ms 2018-09-11 02:08:44,516-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-2) [] Operation Failed: [must be greater than or equal to 1, Attribute: cluster.logMaxMemoryUsedThreshold] 2018-09-11 02:08:44,533-04 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter] (default task-2) [] Entered SsoRestApiAuthFilter Thanks, Dafna

3 12

[JIRA] (OVIRT-2485) shift.ovirt.org tls verification does not seem to work
by Barak Korren (oVirt JIRA) 13 Sep '18

13 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2485?page=com.atlassian.jira.… ] Barak Korren reassigned OVIRT-2485: ----------------------------------- Assignee: Evgheni Dereveanchin (was: infra) > shift.ovirt.org tls verification does not seem to work > ------------------------------------------------------ > > Key: OVIRT-2485 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2485 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Roman Mohr > Assignee: Evgheni Dereveanchin > > Hi, > If I expose an app on shift.ovirt.org via a tls endpoint, I get public > urls like this: https://kubevirt-prow.apps.ovirt.org/hook. I can also > use them e.g. on github for webhook, however I have to disable tls > verification. > For instance github gives me otherwise: > We couldn’t deliver this payload: Peer certificate cannot be > authenticated with given CA certificates > Should there be a verifyable certificate? > Best Regards, > Roman -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2493) Please add Avital Pinnick to CI whitelist
by Barak Korren (oVirt JIRA) 13 Sep '18

13 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2493?page=com.atlassian.jira.… ] Barak Korren reassigned OVIRT-2493: ----------------------------------- Assignee: Dusan Fodor (was: infra) > Please add Avital Pinnick to CI whitelist > ----------------------------------------- > > Key: OVIRT-2493 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2493 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Ondra Machacek > Assignee: Dusan Fodor > > Hi, > please add Avital Pinnick (apinnick) to following projects CI whitelist: > ovirt-engine-api-model > Avital is part of documentation team working on api-model. > [~apinnick(a)redhat.com] > Thanks, > Ondra -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2493) Please add Avital Pinnick to CI whitelist
by Ondra Machacek (oVirt JIRA) 13 Sep '18

13 Sep '18

Ondra Machacek created OVIRT-2493: ------------------------------------- Summary: Please add Avital Pinnick to CI whitelist Key: OVIRT-2493 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2493 Project: oVirt - virtualization made easy Issue Type: Task Reporter: Ondra Machacek Assignee: infra Hi, please add Avital Pinnick (apinnick) to following projects CI whitelist: ovirt-engine-api-model Avital is part of documentation team working on api-model. [~apinnick(a)redhat.com] Thanks, Ondra -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2492) ds-vdsm was not used for rhv CQ tests
by Barak Korren (oVirt JIRA) 13 Sep '18

13 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2492?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-2492: ------------------------------------- And please move this ticket to the DS jira. > ds-vdsm was not used for rhv CQ tests > ------------------------------------- > > Key: OVIRT-2492 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2492 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_failures, ost_packages > > we found that vdsm was downloaded from > http://download-node-02.eng.bos.redhat.com/rel-eng/repos/rhevm-4.2-rhel-7-p… > since its a ds project it should be going down from: > http://ci-web.eng.lab.tlv.redhat.com/repos/tested/rhv-4.2/rpm/el7/ > The reason it was downloaded is because package version numbers in official 4.2 is higher then the one in tested so reposync tool was taking it from the wrong repo. > I am adding patches to exclude official repo so that vdsm will download from tested. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2492) ds-vdsm was not used for rhv CQ tests
by Barak Korren (oVirt JIRA) 13 Sep '18

13 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2492?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-2492: ------------------------------------- Don't exclude, ask [~amarchuk], [~dfodor] or [~ena(a)redhat.com] to tag the DS vdsm repo so that you get the proper version number in the CI build. > ds-vdsm was not used for rhv CQ tests > ------------------------------------- > > Key: OVIRT-2492 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2492 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_failures, ost_packages > > we found that vdsm was downloaded from > http://download-node-02.eng.bos.redhat.com/rel-eng/repos/rhevm-4.2-rhel-7-p… > since its a ds project it should be going down from: > http://ci-web.eng.lab.tlv.redhat.com/repos/tested/rhv-4.2/rpm/el7/ > The reason it was downloaded is because package version numbers in official 4.2 is higher then the one in tested so reposync tool was taking it from the wrong repo. > I am adding patches to exclude official repo so that vdsm will download from tested. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2492) ds-vdsm was not used for rhv CQ tests
by Dafna Ron (oVirt JIRA) 13 Sep '18

13 Sep '18

Dafna Ron created OVIRT-2492: -------------------------------- Summary: ds-vdsm was not used for rhv CQ tests Key: OVIRT-2492 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2492 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Dafna Ron Assignee: infra Priority: Highest we found that vdsm was downloaded from http://download-node-02.eng.bos.redhat.com/rel-eng/repos/rhevm-4.2-rhel-7-p… since its a ds project it should be going down from: http://ci-web.eng.lab.tlv.redhat.com/repos/tested/rhv-4.2/rpm/el7/ The reason it was downloaded is because package version numbers in official 4.2 is higher then the one in tested so reposync tool was taking it from the wrong repo. I am adding patches to exclude official repo so that vdsm will download from tested. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2486) master ovirt-engine is failing for both basic and upgrade suites on test 002_bootstrap.add_cluster
by Dafna Ron (oVirt JIRA) 13 Sep '18

13 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2486?page=com.atlassian.jira.… ] Dafna Ron updated OVIRT-2486: ----------------------------- Resolution: Fixed Status: Done (was: To Do) > master ovirt-engine is failing for both basic and upgrade suites on test 002_bootstrap.add_cluster > -------------------------------------------------------------------------------------------------- > > Key: OVIRT-2486 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2486 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_code_regression, ost_failures > > the failure is due to a regression, > change reported for this failure: https://gerrit.ovirt.org/#/c/93345/ > it seems correct. > error: https://pastebin.com/7x0TanJr > full logs: > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10187/artifa… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2486) master ovirt-engine is failing for both basic and upgrade suites on test 002_bootstrap.add_cluster
by Dafna Ron (oVirt JIRA) 13 Sep '18

13 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2486?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2486: ---------------------------------- closing this as this specific issue is resolved > master ovirt-engine is failing for both basic and upgrade suites on test 002_bootstrap.add_cluster > -------------------------------------------------------------------------------------------------- > > Key: OVIRT-2486 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2486 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_code_regression, ost_failures > > the failure is due to a regression, > change reported for this failure: https://gerrit.ovirt.org/#/c/93345/ > it seems correct. > error: https://pastebin.com/7x0TanJr > full logs: > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10187/artifa… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0