Infra
Threads by month
- ----- 2026 -----
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2025 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2011 -----
- December
- November
- October
- September
- 33379 discussions
oVirt infra daily report - unstable production jobs - 565
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18
27 Aug '18
Good morning!
Attached is the HTML page with the jenkins status report. You can see it also here:
- http://jenkins.ovirt.org/job/system_jenkins-report/565//artifact/exported-a…
Cheers,
Jenkins
1
0
Change 93925,2 (vdsm) is probably the reason behind recent system test failures
in the "ovirt-4.2" change queue and needs to be fixed.
This change had been removed from the testing queue. Artifacts build from this
change will not be released until it is fixed.
For further details about the change see:
https://gerrit.ovirt.org/#/c/93925/2
For failed test results see:
http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2993/
1
0
[oVirt Jenkins] ovirt-system-tests_he-basic-iscsi-suite-master -
Build # 464 - Failure!
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18
27 Aug '18
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master…
Build Number: 464
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #464
[Roy Golan] Introduce openshift-on-ovirt
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: 008_restart_he_vm.restart_he_vm
Error Message:
could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1",
-------------------- >> begin captured logging << --------------------
lago.ssh: DEBUG: start task:a3c71d1c-cb12-4fef-9e6a-d5e0c1b07550:Get ssh client for lago-he-basic-iscsi-suite-master-host-1:
lago.ssh: DEBUG: end task:a3c71d1c-cb12-4fef-9e6a-d5e0c1b07550:Get ssh client for lago-he-basic-iscsi-suite-master-host-1:
lago.ssh: DEBUG: Running 43cfc44a on lago-he-basic-iscsi-suite-master-host-1: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 43cfc44a on lago-he-basic-iscsi-suite-master-host-1 returned with 0
lago.ssh: DEBUG: Command 43cfc44a on lago-he-basic-iscsi-suite-master-host-1 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2805 (Sun Aug 26 23:28:20 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2806 (Sun Aug 26 23:28:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd22cf78", "local_conf_timestamp": 2806, "host-ts": 2805}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2810 (Sun Aug 26 23:28:24 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2812 (Sun Aug 26 23:28:26 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7f7e634d", "local_conf_timestamp": 2812, "host-ts": 2810}, "global_maintenance": true}
root: INFO: * Shutting down HE VM on host: lago-he-basic-iscsi-suite-master-host-0
lago.ssh: DEBUG: start task:42afe970-0be5-40d6-834b-245d877c0d1a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:42afe970-0be5-40d6-834b-245d877c0d1a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 445276b0 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-shutdown
lago.ssh: DEBUG: Command 445276b0 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
root: INFO: * Command succeeded
root: INFO: * Waiting for VM to be down...
lago.ssh: DEBUG: start task:d92ec1bd-a571-4e86-b3a6-7bedd805567d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:d92ec1bd-a571-4e86-b3a6-7bedd805567d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 44f97960 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 44f97960 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 44f97960 on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2805 (Sun Aug 26 23:28:20 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2806 (Sun Aug 26 23:28:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd22cf78", "local_conf_timestamp": 2806, "host-ts": 2805}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2800 (Sun Aug 26 23:28:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2801 (Sun Aug 26 23:28:15 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "063aa920", "local_conf_timestamp": 2801, "host-ts": 2800}, "global_maintenance": false}
lago.ssh: DEBUG: start task:a1e22b81-c1cf-4b25-a1f7-342286fbe81b:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:a1e22b81-c1cf-4b25-a1f7-342286fbe81b:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true}
lago.ssh: DEBUG: start task:f5948091-c497-4f3b-9761-04d42a2cc421:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:f5948091-c497-4f3b-9761-04d42a2cc421:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 4df162c6 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 4df162c6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 4df162c6 on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true}
lago.ssh: DEBUG: start task:8b462547-686e-4dbe-8bfb-91d4ba2931dd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:8b462547-686e-4dbe-8bfb-91d4ba2931dd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 504ec626 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 504ec626 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 504ec626 on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true}
lago.ssh: DEBUG: start task:f2eef89a-f42e-4c79-9337-72571575f679:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:f2eef89a-f42e-4c79-9337-72571575f679:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 52a4916c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 52a4916c on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 52a4916c on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2835 (Sun Aug 26 23:28:50 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2837 (Sun Aug 26 23:28:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7a4710c6", "local_conf_timestamp": 2837, "host-ts": 2835}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2831 (Sun Aug 26 23:28:45 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2833 (Sun Aug 26 23:28:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fce69ba8", "local_conf_timestamp": 2833, "host-ts": 2831}, "global_maintenance": true}
root: INFO: * VM is down.
root: INFO: * Stopping services...
lago.ssh: DEBUG: start task:2c25f4a6-f2ef-4cfa-822f-b929c4dfd509:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:2c25f4a6-f2ef-4cfa-822f-b929c4dfd509:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 5322b5b0 on lago-he-basic-iscsi-suite-master-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent
lago.ssh: DEBUG: Command 5322b5b0 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
root: INFO: * Starting services...
lago.ssh: DEBUG: start task:ee6b4609-b5ac-4f3c-a182-7ed44b8deeb6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:ee6b4609-b5ac-4f3c-a182-7ed44b8deeb6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 5a5b1dd6 on lago-he-basic-iscsi-suite-master-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent
lago.ssh: DEBUG: Command 5a5b1dd6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
root: INFO: * Waiting for agent to be ready...
lago.ssh: DEBUG: start task:facd8e55-61ed-4fb7-b9fe-c3ad705d968a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:facd8e55-61ed-4fb7-b9fe-c3ad705d968a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0 returned with 1
lago.ssh: DEBUG: Command 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:3eb14b7c-074b-4c11-a117-392f8b162c2e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:3eb14b7c-074b-4c11-a117-392f8b162c2e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 62a124ea on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command 62a124ea on lago-he-basic-iscsi-suite-master-host-0 returned with 1
lago.ssh: DEBUG: Command 62a124ea on lago-he-basic-iscsi-suite-master-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:c78b3a53-f7bb-480e-839e-b57e9ca7eb4d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:c78b3a53-f7bb-480e-839e-b57e9ca7eb4d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0 returned with 1
lago.ssh: DEBUG: Command 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:ca525d74-b2b6-4297-ad54-b044bc452568:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:ca525d74-b2b6-4297-ad54-b044bc452568:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 66d48ffc on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command 66d48ffc on lago-he-basic-iscsi-suite-master-host-0 returned with 1
lago.ssh: DEBUG: Command 66d48ffc on lago-he-basic-iscsi-suite-master-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:f871d6fb-e79f-477b-b5ca-99b939a4d9f7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:f871d6fb-e79f-477b-b5ca-99b939a4d9f7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 69018c26 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command 69018c26 on lago-he-basic-iscsi-suite-master-host-0 returned with 1
lago.ssh: DEBUG: Command 69018c26 on lago-he-basic-iscsi-suite-master-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:b86743b4-04cf-4455-aba9-0bc61ad3815e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:b86743b4-04cf-4455-aba9-0bc61ad3815e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0 returned with 1
lago.ssh: DEBUG: Command 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:8e87c593-e4ec-4db5-b533-eab312a2c1cb:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:8e87c593-e4ec-4db5-b533-eab312a2c1cb:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 6d468714 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command 6d468714 on lago-he-basic-iscsi-suite-master-host-0 returned with 1
lago.ssh: DEBUG: Command 6d468714 on lago-he-basic-iscsi-suite-master-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:221b1e18-be90-427f-8a54-9291fe6f5959:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:221b1e18-be90-427f-8a54-9291fe6f5959:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 6f693f32 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command 6f693f32 on lago-he-basic-iscsi-suite-master-host-0 returned with 1
lago.ssh: DEBUG: Command 6f693f32 on lago-he-basic-iscsi-suite-master-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:52fbd8f8-8d38-4465-9ecc-c00e381b7563:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:52fbd8f8-8d38-4465-9ecc-c00e381b7563:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 71a1db42 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command 71a1db42 on lago-he-basic-iscsi-suite-master-host-0 returned with 1
lago.ssh: DEBUG: Command 71a1db42 on lago-he-basic-iscsi-suite-master-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:7a8d4a8c-12b5-4a68-b436-535b8d09f517:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:7a8d4a8c-12b5-4a68-b436-535b8d09f517:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0 returned with 1
lago.ssh: DEBUG: Command 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0 output:
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
lago.ssh: DEBUG: start task:3de81b6a-badb-4222-8aca-128ebdfec400:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:3de81b6a-badb-4222-8aca-128ebdfec400:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 75dd4066 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status
lago.ssh: DEBUG: Command 75dd4066 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 75dd4066 on lago-he-basic-iscsi-suite-master-host-0 output:
!! Cluster is in GLOBAL MAINTENANCE mode !!
--== Host lago-he-basic-iscsi-suite-master-host-0.lago.local (id: 1) status ==--
conf_on_shared_storage : True
Status up-to-date : True
Hostname : lago-he-basic-iscsi-suite-master-host-0.lago.local
Host ID : 1
Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}
Score : 0
stopped : False
Local maintenance : False
crc32 : 56f93df8
local_conf_timestamp : 2896
Host timestamp : 2895
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=2895 (Sun Aug 26 23:29:50 2018)
host-id=1
score=0
vm_conf_refresh_time=2896 (Sun Aug 26 23:29:51 2018)
conf_on_shared_storage=True
maintenance=False
state=ReinitializeFSM
stopped=False
--== Host lago-he-basic-iscsi-suite-master-host-1 (id: 2) status ==--
conf_on_shared_storage : True
Status up-to-date : False
Hostname : lago-he-basic-iscsi-suite-master-host-1
Host ID : 2
Engine status : unknown stale-data
Score : 3400
stopped : False
Local maintenance : False
crc32 : ffa35d6d
local_conf_timestamp : 2892
Host timestamp : 2891
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=2891 (Sun Aug 26 23:29:45 2018)
host-id=2
score=3400
vm_conf_refresh_time=2892 (Sun Aug 26 23:29:46 2018)
conf_on_shared_storage=True
maintenance=False
state=GlobalMaintenance
stopped=False
!! Cluster is in GLOBAL MAINTENANCE mode !!
root: INFO: * Agent is ready.
root: INFO: * Starting VM...
lago.ssh: DEBUG: start task:0174ef46-3f09-4c57-b0cc-135fc059665a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:0174ef46-3f09-4c57-b0cc-135fc059665a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 764e856e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-start
lago.ssh: DEBUG: Command 764e856e on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 764e856e on lago-he-basic-iscsi-suite-master-host-0 output:
VM exists and is down, cleaning up and restarting
root: INFO: * Command succeeded
root: INFO: * Waiting for VM to be UP...
lago.ssh: DEBUG: start task:30ee44ce-7f0b-4b16-ba1b-7b6964891efd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:30ee44ce-7f0b-4b16-ba1b-7b6964891efd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 77b7ccda on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 77b7ccda on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 77b7ccda on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2895 (Sun Aug 26 23:29:50 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=2896 (Sun Aug 26 23:29:51 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=ReinitializeFSM\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "56f93df8", "local_conf_timestamp": 2896, "host-ts": 2895}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2891 (Sun Aug 26 23:29:45 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2892 (Sun Aug 26 23:29:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ffa35d6d", "local_conf_timestamp": 2892, "host-ts": 2891}, "global_maintenance": true}
lago.ssh: DEBUG: start task:ef26fadc-fd32-4afb-9a06-07e13a3e556f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:ef26fadc-fd32-4afb-9a06-07e13a3e556f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2905 (Sun Aug 26 23:30:00 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2906 (Sun Aug 26 23:30:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9fce1b92", "local_conf_timestamp": 2906, "host-ts": 2905}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2901 (Sun Aug 26 23:29:55 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2902 (Sun Aug 26 23:29:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "68110bea", "local_conf_timestamp": 2902, "host-ts": 2901}, "global_maintenance": true}
root: INFO: * VM is UP.
root: INFO: * Waiting for engine to start...
lago.ssh: DEBUG: start task:22f9ce85-5236-43de-956e-60eedabdee44:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:22f9ce85-5236-43de-956e-60eedabdee44:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2905 (Sun Aug 26 23:30:00 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2906 (Sun Aug 26 23:30:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9fce1b92", "local_conf_timestamp": 2906, "host-ts": 2905}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2901 (Sun Aug 26 23:29:55 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2902 (Sun Aug 26 23:29:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "68110bea", "local_conf_timestamp": 2902, "host-ts": 2901}, "global_maintenance": true}
lago.ssh: DEBUG: start task:10d11112-af95-4062-82b5-90a12379d806:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:10d11112-af95-4062-82b5-90a12379d806:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 85234b10 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 85234b10 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 85234b10 on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2915 (Sun Aug 26 23:30:10 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2916 (Sun Aug 26 23:30:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69a0d34f", "local_conf_timestamp": 2916, "host-ts": 2915}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2911 (Sun Aug 26 23:30:05 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2913 (Sun Aug 26 23:30:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ff5ba0ab", "local_conf_timestamp": 2913, "host-ts": 2911}, "global_maintenance": true}
lago.ssh: DEBUG: start task:8600ff29-4795-4922-9bb5-1ea8cb47aff6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:8600ff29-4795-4922-9bb5-1ea8cb47aff6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 8778cab6 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 8778cab6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 8778cab6 on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2915 (Sun Aug 26 23:30:10 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2916 (Sun Aug 26 23:30:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69a0d34f", "local_conf_timestamp": 2916, "host-ts": 2915}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true}
lago.ssh: DEBUG: start task:6251e2ec-36ea-42cd-a7cf-018d3efaf73a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:6251e2ec-36ea-42cd-a7cf-018d3efaf73a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 89e10548 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 89e10548 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 89e10548 on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2925 (Sun Aug 26 23:30:20 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2927 (Sun Aug 26 23:30:22 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "df65bcff", "local_conf_timestamp": 2927, "host-ts": 2925}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true}
lago.ssh: DEBUG: start task:5671a60c-09fc-4f60-97a2-270f9f5b4ca1:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:5671a60c-09fc-4f60-97a2-270f9f5b4ca1:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2925 (Sun Aug 26 23:30:20 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2927 (Sun Aug 26 23:30:22 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "df65bcff", "local_conf_timestamp": 2927, "host-ts": 2925}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true}
lago.ssh: DEBUG: start task:ca12f761-8a01-4613-8217-3848165adb82:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:ca12f761-8a01-4613-8217-3848165adb82:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 8efac10e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 8efac10e on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 8efac10e on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2935 (Sun Aug 26 23:30:30 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2937 (Sun Aug 26 23:30:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "41a00e65", "local_conf_timestamp": 2937, "host-ts": 2935}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2932 (Sun Aug 26 23:30:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2934 (Sun Aug 26 23:30:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "903a0802", "local_conf_timestamp": 2934, "host-ts": 2932}, "global_maintenance": true}
lago.ssh: DEBUG: start task:2ce2a039-af5f-47e2-b0c8-650b8ca40276:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:2ce2a039-af5f-47e2-b0c8-650b8ca40276:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2935 (Sun Aug 26 23:30:30 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2937 (Sun Aug 26 23:30:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "41a00e65", "local_conf_timestamp": 2937, "host-ts": 2935}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2932 (Sun Aug 26 23:30:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2934 (Sun Aug 26 23:30:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "903a0802", "local_conf_timestamp": 2934, "host-ts": 2932}, "global_maintenance": true}
lago.ssh: DEBUG: start task:50565f8c-6b24-4ae7-9eba-bbf9b31b3867:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:50565f8c-6b24-4ae7-9eba-bbf9b31b3867:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 93ca2012 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 93ca2012 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 93ca2012 on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true}
lago.ssh: DEBUG: start task:8de05a79-ab0e-4659-bc5c-03ab090af777:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:8de05a79-ab0e-4659-bc5c-03ab090af777:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 964c95f4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 964c95f4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 964c95f4 on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true}
lago.ssh: DEBUG: start task:c85287ef-a0b5-4cea-ab1a-cfc1c4cc97d4:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:c85287ef-a0b5-4cea-ab1a-cfc1c4cc97d4:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 98aaf00c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 98aaf00c on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 98aaf00c on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true}
lago.ssh: DEBUG: start task:c1aa3391-00ad-4c4a-a698-304aa8d8e598:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:c1aa3391-00ad-4c4a-a698-304aa8d8e598:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 9b10536e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 9b10536e on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 9b10536e on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2955 (Sun Aug 26 23:30:50 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2957 (Sun Aug 26 23:30:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6d9e5c54", "local_conf_timestamp": 2957, "host-ts": 2955}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2952 (Sun Aug 26 23:30:46 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2953 (Sun Aug 26 23:30:47 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2260cf90", "local_conf_timestamp": 2953, "host-ts": 2952}, "global_maintenance": true}
lago.ssh: DEBUG: start task:a7f2e7dd-ffdc-4265-a6b0-58c16e4d076f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:a7f2e7dd-ffdc-4265-a6b0-58c16e4d076f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 9d796f1e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 9d796f1e on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 9d796f1e on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2955 (Sun Aug 26 23:30:50 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2957 (Sun Aug 26 23:30:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6d9e5c54", "local_conf_timestamp": 2957, "host-ts": 2955}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2952 (Sun Aug 26 23:30:46 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2953 (Sun Aug 26 23:30:47 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2260cf90", "local_conf_timestamp": 2953, "host-ts": 2952}, "global_maintenance": true}
lago.ssh: DEBUG: start task:197b6456-4573-4fe5-a902-4d4135afd1c5:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:197b6456-4573-4fe5-a902-4d4135afd1c5:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running 9fef209a on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 9fef209a on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command 9fef209a on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true}
lago.ssh: DEBUG: start task:e449730a-24b3-4ba7-a32a-9841d681e8d7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:e449730a-24b3-4ba7-a32a-9841d681e8d7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running a251ea20 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command a251ea20 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command a251ea20 on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true}
lago.ssh: DEBUG: start task:8fa960e8-d3d3-4de3-8673-7ccb2e8e3232:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:8fa960e8-d3d3-4de3-8673-7ccb2e8e3232:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running a4b99e3e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command a4b99e3e on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command a4b99e3e on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true}
lago.ssh: DEBUG: start task:df444f3d-0958-4232-bbe2-79c9b53d6df7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:df444f3d-0958-4232-bbe2-79c9b53d6df7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running a7386262 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command a7386262 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command a7386262 on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2975 (Sun Aug 26 23:31:10 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2976 (Sun Aug 26 23:31:11 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1cb13a70", "local_conf_timestamp": 2976, "host-ts": 2975}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2972 (Sun Aug 26 23:31:06 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2973 (Sun Aug 26 23:31:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "398afe7f", "local_conf_timestamp": 2973, "host-ts": 2972}, "global_maintenance": true}
lago.ssh: DEBUG: start task:ccb637f6-e93c-474f-8822-99fa3e47e492:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:ccb637f6-e93c-474f-8822-99fa3e47e492:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2975 (Sun Aug 26 23:31:10 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2976 (Sun Aug 26 23:31:11 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1cb13a70", "local_conf_timestamp": 2976, "host-ts": 2975}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2972 (Sun Aug 26 23:31:06 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2973 (Sun Aug 26 23:31:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "398afe7f", "local_conf_timestamp": 2973, "host-ts": 2972}, "global_maintenance": true}
lago.ssh: DEBUG: start task:3c133bbf-6ccf-4f97-a708-4902c159a68f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:3c133bbf-6ccf-4f97-a708-4902c159a68f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running ac14d18a on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command ac14d18a on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command ac14d18a on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true}
lago.ssh: DEBUG: start task:ff5cd16b-0c57-4df9-bbcc-9fd94b5fccb8:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:ff5cd16b-0c57-4df9-bbcc-9fd94b5fccb8:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true}
lago.ssh: DEBUG: start task:7fcd4b38-7d15-4000-ad18-d363a3d3a7ab:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:7fcd4b38-7d15-4000-ad18-d363a3d3a7ab:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running b0c63368 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command b0c63368 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command b0c63368 on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true}
lago.ssh: DEBUG: start task:3c20a1c0-ec61-41b6-b247-25ac0093a494:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:3c20a1c0-ec61-41b6-b247-25ac0093a494:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running b30f7738 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command b30f7738 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command b30f7738 on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7a1c6bf2", "local_conf_timestamp": 2993, "host-ts": 2992}, "global_maintenance": true}
lago.ssh: DEBUG: start task:90657a8e-cbc1-4061-823a-c2e3d71635c3:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: end task:90657a8e-cbc1-4061-823a-c2e3d71635c3:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:
lago.ssh: DEBUG: Running b555e324 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command b555e324 on lago-he-basic-iscsi-suite-master-host-0 returned with 0
lago.ssh: DEBUG: Command b555e324 on lago-he-basic-iscsi-suite-master-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1",
ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f5d8fad1410>
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within
res = func()
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda>
for k, v in _get_he_status(host).items()
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status
raise RuntimeError('could not parse JSON: %s' % ret.out)
RuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1",
--------------------- >> end captured logging << ---------------------
Stack Trace:
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 53, in restart_he_vm
_wait_for_engine_health(host)
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 185, in _wait_for_engine_health
testlib.assert_true_within_long(lambda: any(
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long
assert_equals_within_long(func, True, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long
func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within
res = func()
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda>
for k, v in _get_he_status(host).items()
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status
raise RuntimeError('could not parse JSON: %s' % ret.out)
'could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1",\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:a3c71d1c-cb12-4fef-9e6a-d5e0c1b07550:Get ssh client for lago-he-basic-iscsi-suite-master-host-1:\nlago.ssh: DEBUG: end task:a3c71d1c-cb12-4fef-9e6a-d5e0c1b07550:Get ssh client for lago-he-basic-iscsi-suite-master-host-1:\nlago.ssh: DEBUG: Running 43cfc44a on lago-he-basic-iscsi-suite-master-host-1: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 43cfc44a on lago-he-basic-iscsi-suite-master-host-1 returned with 0\nlago.ssh: DEBUG: Command 43cfc44a on lago-he-basic-iscsi-suite-master-host-1 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2805 (Sun Aug 26 23:28:20 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2806 (Sun Aug 26 23:28:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd22cf78", "local_conf_timestamp": 2806, "host-ts": 2805}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2810 (Sun Aug 26 23:28:24 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2812 (Sun Aug 26 23:28:26 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7f7e634d", "local_conf_timestamp": 2812, "host-ts": 2810}, "global_maintenance": true}\n\nroot: INFO: * Shutting down HE VM on host: lago-he-basic-iscsi-suite-master-host-0\nlago.ssh: DEBUG: start task:42afe970-0be5-40d6-834b-245d877c0d1a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:42afe970-0be5-40d6-834b-245d877c0d1a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 445276b0 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-shutdown\nlago.ssh: DEBUG: Command 445276b0 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be down...\nlago.ssh: DEBUG: start task:d92ec1bd-a571-4e86-b3a6-7bedd805567d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:d92ec1bd-a571-4e86-b3a6-7bedd805567d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 44f97960 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 44f97960 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 44f97960 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2805 (Sun Aug 26 23:28:20 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2806 (Sun Aug 26 23:28:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd22cf78", "local_conf_timestamp": 2806, "host-ts": 2805}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2800 (Sun Aug 26 23:28:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2801 (Sun Aug 26 23:28:15 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "063aa920", "local_conf_timestamp": 2801, "host-ts": 2800}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:a1e22b81-c1cf-4b25-a1f7-342286fbe81b:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:a1e22b81-c1cf-4b25-a1f7-342286fbe81b:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:f5948091-c497-4f3b-9761-04d42a2cc421:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:f5948091-c497-4f3b-9761-04d42a2cc421:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 4df162c6 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 4df162c6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 4df162c6 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8b462547-686e-4dbe-8bfb-91d4ba2931dd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:8b462547-686e-4dbe-8bfb-91d4ba2931dd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 504ec626 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 504ec626 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 504ec626 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:f2eef89a-f42e-4c79-9337-72571575f679:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:f2eef89a-f42e-4c79-9337-72571575f679:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 52a4916c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 52a4916c on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 52a4916c on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2835 (Sun Aug 26 23:28:50 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2837 (Sun Aug 26 23:28:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7a4710c6", "local_conf_timestamp": 2837, "host-ts": 2835}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2831 (Sun Aug 26 23:28:45 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2833 (Sun Aug 26 23:28:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fce69ba8", "local_conf_timestamp": 2833, "host-ts": 2831}, "global_maintenance": true}\n\nroot: INFO: * VM is down.\nroot: INFO: * Stopping services...\nlago.ssh: DEBUG: start task:2c25f4a6-f2ef-4cfa-822f-b929c4dfd509:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:2c25f4a6-f2ef-4cfa-822f-b929c4dfd509:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 5322b5b0 on lago-he-basic-iscsi-suite-master-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command 5322b5b0 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nroot: INFO: * Starting services...\nlago.ssh: DEBUG: start task:ee6b4609-b5ac-4f3c-a182-7ed44b8deeb6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ee6b4609-b5ac-4f3c-a182-7ed44b8deeb6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 5a5b1dd6 on lago-he-basic-iscsi-suite-master-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command 5a5b1dd6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nroot: INFO: * Waiting for agent to be ready...\nlago.ssh: DEBUG: start task:facd8e55-61ed-4fb7-b9fe-c3ad705d968a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:facd8e55-61ed-4fb7-b9fe-c3ad705d968a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:3eb14b7c-074b-4c11-a117-392f8b162c2e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:3eb14b7c-074b-4c11-a117-392f8b162c2e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 62a124ea on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 62a124ea on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 62a124ea on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:c78b3a53-f7bb-480e-839e-b57e9ca7eb4d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:c78b3a53-f7bb-480e-839e-b57e9ca7eb4d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:ca525d74-b2b6-4297-ad54-b044bc452568:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ca525d74-b2b6-4297-ad54-b044bc452568:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 66d48ffc on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 66d48ffc on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 66d48ffc on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:f871d6fb-e79f-477b-b5ca-99b939a4d9f7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:f871d6fb-e79f-477b-b5ca-99b939a4d9f7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 69018c26 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 69018c26 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 69018c26 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:b86743b4-04cf-4455-aba9-0bc61ad3815e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:b86743b4-04cf-4455-aba9-0bc61ad3815e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:8e87c593-e4ec-4db5-b533-eab312a2c1cb:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:8e87c593-e4ec-4db5-b533-eab312a2c1cb:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 6d468714 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 6d468714 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 6d468714 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:221b1e18-be90-427f-8a54-9291fe6f5959:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:221b1e18-be90-427f-8a54-9291fe6f5959:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 6f693f32 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 6f693f32 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 6f693f32 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:52fbd8f8-8d38-4465-9ecc-c00e381b7563:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:52fbd8f8-8d38-4465-9ecc-c00e381b7563:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 71a1db42 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 71a1db42 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 71a1db42 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:7a8d4a8c-12b5-4a68-b436-535b8d09f517:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:7a8d4a8c-12b5-4a68-b436-535b8d09f517:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:3de81b6a-badb-4222-8aca-128ebdfec400:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:3de81b6a-badb-4222-8aca-128ebdfec400:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 75dd4066 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 75dd4066 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 75dd4066 on lago-he-basic-iscsi-suite-master-host-0 output:\n \n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\n\n--== Host lago-he-basic-iscsi-suite-master-host-0.lago.local (id: 1) status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : True\nHostname : lago-he-basic-iscsi-suite-master-host-0.lago.local\nHost ID : 1\nEngine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}\nScore : 0\nstopped : False\nLocal maintenance : False\ncrc32 : 56f93df8\nlocal_conf_timestamp : 2896\nHost timestamp : 2895\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2895 (Sun Aug 26 23:29:50 2018)\n\thost-id=1\n\tscore=0\n\tvm_conf_refresh_time=2896 (Sun Aug 26 23:29:51 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=ReinitializeFSM\n\tstopped=False\n\n\n--== Host lago-he-basic-iscsi-suite-master-host-1 (id: 2) status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : False\nHostname : lago-he-basic-iscsi-suite-master-host-1\nHost ID : 2\nEngine status : unknown stale-data\nScore : 3400\nstopped : False\nLocal maintenance : False\ncrc32 : ffa35d6d\nlocal_conf_timestamp : 2892\nHost timestamp : 2891\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2891 (Sun Aug 26 23:29:45 2018)\n\thost-id=2\n\tscore=3400\n\tvm_conf_refresh_time=2892 (Sun Aug 26 23:29:46 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=GlobalMaintenance\n\tstopped=False\n\n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\nroot: INFO: * Agent is ready.\nroot: INFO: * Starting VM...\nlago.ssh: DEBUG: start task:0174ef46-3f09-4c57-b0cc-135fc059665a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:0174ef46-3f09-4c57-b0cc-135fc059665a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 764e856e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-start\nlago.ssh: DEBUG: Command 764e856e on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 764e856e on lago-he-basic-iscsi-suite-master-host-0 output:\n VM exists and is down, cleaning up and restarting\n\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be UP...\nlago.ssh: DEBUG: start task:30ee44ce-7f0b-4b16-ba1b-7b6964891efd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:30ee44ce-7f0b-4b16-ba1b-7b6964891efd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 77b7ccda on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 77b7ccda on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 77b7ccda on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2895 (Sun Aug 26 23:29:50 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=2896 (Sun Aug 26 23:29:51 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=ReinitializeFSM\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "56f93df8", "local_conf_timestamp": 2896, "host-ts": 2895}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2891 (Sun Aug 26 23:29:45 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2892 (Sun Aug 26 23:29:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ffa35d6d", "local_conf_timestamp": 2892, "host-ts": 2891}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ef26fadc-fd32-4afb-9a06-07e13a3e556f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ef26fadc-fd32-4afb-9a06-07e13a3e556f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2905 (Sun Aug 26 23:30:00 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2906 (Sun Aug 26 23:30:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9fce1b92", "local_conf_timestamp": 2906, "host-ts": 2905}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2901 (Sun Aug 26 23:29:55 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2902 (Sun Aug 26 23:29:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "68110bea", "local_conf_timestamp": 2902, "host-ts": 2901}, "global_maintenance": true}\n\nroot: INFO: * VM is UP.\nroot: INFO: * Waiting for engine to start...\nlago.ssh: DEBUG: start task:22f9ce85-5236-43de-956e-60eedabdee44:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:22f9ce85-5236-43de-956e-60eedabdee44:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2905 (Sun Aug 26 23:30:00 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2906 (Sun Aug 26 23:30:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9fce1b92", "local_conf_timestamp": 2906, "host-ts": 2905}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2901 (Sun Aug 26 23:29:55 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2902 (Sun Aug 26 23:29:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "68110bea", "local_conf_timestamp": 2902, "host-ts": 2901}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:10d11112-af95-4062-82b5-90a12379d806:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:10d11112-af95-4062-82b5-90a12379d806:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 85234b10 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 85234b10 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 85234b10 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2915 (Sun Aug 26 23:30:10 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2916 (Sun Aug 26 23:30:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69a0d34f", "local_conf_timestamp": 2916, "host-ts": 2915}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2911 (Sun Aug 26 23:30:05 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2913 (Sun Aug 26 23:30:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ff5ba0ab", "local_conf_timestamp": 2913, "host-ts": 2911}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8600ff29-4795-4922-9bb5-1ea8cb47aff6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:8600ff29-4795-4922-9bb5-1ea8cb47aff6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 8778cab6 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 8778cab6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 8778cab6 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2915 (Sun Aug 26 23:30:10 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2916 (Sun Aug 26 23:30:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69a0d34f", "local_conf_timestamp": 2916, "host-ts": 2915}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:6251e2ec-36ea-42cd-a7cf-018d3efaf73a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:6251e2ec-36ea-42cd-a7cf-018d3efaf73a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 89e10548 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 89e10548 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 89e10548 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2925 (Sun Aug 26 23:30:20 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2927 (Sun Aug 26 23:30:22 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "df65bcff", "local_conf_timestamp": 2927, "host-ts": 2925}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:5671a60c-09fc-4f60-97a2-270f9f5b4ca1:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:5671a60c-09fc-4f60-97a2-270f9f5b4ca1:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2925 (Sun Aug 26 23:30:20 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2927 (Sun Aug 26 23:30:22 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "df65bcff", "local_conf_timestamp": 2927, "host-ts": 2925}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ca12f761-8a01-4613-8217-3848165adb82:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ca12f761-8a01-4613-8217-3848165adb82:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 8efac10e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 8efac10e on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 8efac10e on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2935 (Sun Aug 26 23:30:30 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2937 (Sun Aug 26 23:30:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "41a00e65", "local_conf_timestamp": 2937, "host-ts": 2935}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2932 (Sun Aug 26 23:30:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2934 (Sun Aug 26 23:30:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "903a0802", "local_conf_timestamp": 2934, "host-ts": 2932}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:2ce2a039-af5f-47e2-b0c8-650b8ca40276:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:2ce2a039-af5f-47e2-b0c8-650b8ca40276:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2935 (Sun Aug 26 23:30:30 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2937 (Sun Aug 26 23:30:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "41a00e65", "local_conf_timestamp": 2937, "host-ts": 2935}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2932 (Sun Aug 26 23:30:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2934 (Sun Aug 26 23:30:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "903a0802", "local_conf_timestamp": 2934, "host-ts": 2932}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:50565f8c-6b24-4ae7-9eba-bbf9b31b3867:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:50565f8c-6b24-4ae7-9eba-bbf9b31b3867:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 93ca2012 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 93ca2012 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 93ca2012 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8de05a79-ab0e-4659-bc5c-03ab090af777:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:8de05a79-ab0e-4659-bc5c-03ab090af777:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 964c95f4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 964c95f4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 964c95f4 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:c85287ef-a0b5-4cea-ab1a-cfc1c4cc97d4:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:c85287ef-a0b5-4cea-ab1a-cfc1c4cc97d4:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 98aaf00c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 98aaf00c on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 98aaf00c on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:c1aa3391-00ad-4c4a-a698-304aa8d8e598:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:c1aa3391-00ad-4c4a-a698-304aa8d8e598:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 9b10536e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9b10536e on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 9b10536e on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2955 (Sun Aug 26 23:30:50 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2957 (Sun Aug 26 23:30:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6d9e5c54", "local_conf_timestamp": 2957, "host-ts": 2955}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2952 (Sun Aug 26 23:30:46 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2953 (Sun Aug 26 23:30:47 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2260cf90", "local_conf_timestamp": 2953, "host-ts": 2952}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:a7f2e7dd-ffdc-4265-a6b0-58c16e4d076f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:a7f2e7dd-ffdc-4265-a6b0-58c16e4d076f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 9d796f1e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9d796f1e on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 9d796f1e on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2955 (Sun Aug 26 23:30:50 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2957 (Sun Aug 26 23:30:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6d9e5c54", "local_conf_timestamp": 2957, "host-ts": 2955}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2952 (Sun Aug 26 23:30:46 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2953 (Sun Aug 26 23:30:47 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2260cf90", "local_conf_timestamp": 2953, "host-ts": 2952}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:197b6456-4573-4fe5-a902-4d4135afd1c5:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:197b6456-4573-4fe5-a902-4d4135afd1c5:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 9fef209a on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9fef209a on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 9fef209a on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:e449730a-24b3-4ba7-a32a-9841d681e8d7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:e449730a-24b3-4ba7-a32a-9841d681e8d7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running a251ea20 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a251ea20 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a251ea20 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8fa960e8-d3d3-4de3-8673-7ccb2e8e3232:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:8fa960e8-d3d3-4de3-8673-7ccb2e8e3232:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running a4b99e3e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a4b99e3e on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a4b99e3e on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:df444f3d-0958-4232-bbe2-79c9b53d6df7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:df444f3d-0958-4232-bbe2-79c9b53d6df7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running a7386262 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a7386262 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a7386262 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2975 (Sun Aug 26 23:31:10 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2976 (Sun Aug 26 23:31:11 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1cb13a70", "local_conf_timestamp": 2976, "host-ts": 2975}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2972 (Sun Aug 26 23:31:06 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2973 (Sun Aug 26 23:31:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "398afe7f", "local_conf_timestamp": 2973, "host-ts": 2972}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ccb637f6-e93c-474f-8822-99fa3e47e492:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ccb637f6-e93c-474f-8822-99fa3e47e492:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2975 (Sun Aug 26 23:31:10 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2976 (Sun Aug 26 23:31:11 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1cb13a70", "local_conf_timestamp": 2976, "host-ts": 2975}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2972 (Sun Aug 26 23:31:06 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2973 (Sun Aug 26 23:31:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "398afe7f", "local_conf_timestamp": 2973, "host-ts": 2972}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:3c133bbf-6ccf-4f97-a708-4902c159a68f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:3c133bbf-6ccf-4f97-a708-4902c159a68f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running ac14d18a on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ac14d18a on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ac14d18a on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ff5cd16b-0c57-4df9-bbcc-9fd94b5fccb8:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ff5cd16b-0c57-4df9-bbcc-9fd94b5fccb8:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:7fcd4b38-7d15-4000-ad18-d363a3d3a7ab:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:7fcd4b38-7d15-4000-ad18-d363a3d3a7ab:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running b0c63368 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b0c63368 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command b0c63368 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:3c20a1c0-ec61-41b6-b247-25ac0093a494:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:3c20a1c0-ec61-41b6-b247-25ac0093a494:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running b30f7738 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b30f7738 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command b30f7738 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7a1c6bf2", "local_conf_timestamp": 2993, "host-ts": 2992}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:90657a8e-cbc1-4061-823a-c2e3d71635c3:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:90657a8e-cbc1-4061-823a-c2e3d71635c3:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running b555e324 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b555e324 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command b555e324 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1",\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f5d8fad1410>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda>\n for k, v in _get_he_status(host).items()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status\n raise RuntimeError(\'could not parse JSON: %s\' % ret.out)\nRuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1",\n--------------------- >> end captured logging << ---------------------'
1
2
Build failed in Jenkins:
system-sync_mirrors-centos-updates-el7-x86_64 #1884
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18
27 Aug '18
See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…>
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…>
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 3afc9ba57755a1b2574f7a177184eb66d5f6a90d (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 3afc9ba57755a1b2574f7a177184eb66d5f6a90d
Commit message: "ovirt-release: drop non working jobs"
> git rev-list --no-walk 3afc9ba57755a1b2574f7a177184eb66d5f6a90d # timeout=10
[system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins1863032023757664663.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf
Checking if mirror needs a resync
Traceback (most recent call last):
File "/usr/bin/reposync", line 343, in <module>
main()
File "/usr/bin/reposync", line 175, in main
my.doRepoSetup()
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
return self._getRepos(thisrepo, True)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
self._repos.doSetup(thisrepo)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
dl = repo._async and repo._commonLoadRepoXML(repo)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML
local = self.cachedir + '/repomd.xml'
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda>
cachedir = property(lambda self: self._dirGetAttr('cachedir'))
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr
self.dirSetup()
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup
self._dirSetupMkdir_p(dir)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p
raise Errors.RepoError, msg
yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7'
Build step 'Execute shell' marked build as failure
1
1
[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build #
639 - Failure!
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18
27 Aug '18
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/639/
Build Number: 639
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #639
[Roy Golan] Introduce openshift-on-ovirt
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: 004_basic_sanity.snapshot_merge
Error Message:
status: 409
reason: Conflict
detail: Cannot create Snapshot. Snapshot is currently being created for VM vm0.
Stack Trace:
Traceback (most recent call last):
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 69, in wrapper
return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs)
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/004_basic_sanity.py", line 194, in snapshot_merge
api.vms.get(VM0_NAME).snapshots.add(dead_snap2_params)
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 34511, in add
headers={"Correlation-Id":correlation_id, "Expect":expect}
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add
return self.request('POST', url, body, headers, cls=cls)
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request
persistent_auth=self.__persistent_auth
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request
persistent_auth)
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request
raise errors.RequestError(response_code, response_reason, response_body)
RequestError:
status: 409
reason: Conflict
detail: Cannot create Snapshot. Snapshot is currently being created for VM vm0.
1
1
Change 93756,2 (imgbased) is probably the reason behind recent system test
failures in the "ovirt-4.2" change queue and needs to be fixed.
This change had been removed from the testing queue. Artifacts build from this
change will not be released until it is fixed.
For further details about the change see:
https://gerrit.ovirt.org/#/c/93756/2
For failed test results see:
http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2986/
1
0
[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-4.2 - Build #
255 - Failure!
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18
27 Aug '18
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/255/
Build Number: 255
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #255
[Roy Golan] Introduce openshift-on-ovirt
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: 012_local_maintenance_sdk.local_maintenance
Error Message:
The response content type 'text/html; charset=iso-8859-1' isn't the expected XML
-------------------- >> begin captured logging << --------------------
root: INFO: * Waiting For System Stability...
--------------------- >> end captured logging << ---------------------
Stack Trace:
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper
prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/012_local_maintenance_sdk.py", line 45, in local_maintenance
vm_service = test_utils.get_vm_service(engine, VM_HE_NAME)
File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/test_utils/__init__.py", line 71, in __call__
value = self.func(*args)
File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/test_utils/__init__.py", line 101, in get_vm_service
vm = vms_service.list(search='name={}'.format(vm_name))[0]
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 34154, in list
return self._internal_get(headers, query, wait)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get
return future.wait() if wait else future
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait
return self._code(response)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback
self._check_fault(response)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault
body = self._internal_read_body(response)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body
self._connection.check_xml_content_type(response)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type
response.headers
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type
raise Error(msg)
"The response content type 'text/html; charset=iso-8859-1' isn't the expected XML\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\n--------------------- >> end captured logging << ---------------------"
1
1
[oVirt Jenkins] ovirt-appliance_master_build-artifacts-fc28-x86_64
- Build # 40 - Failure!
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18
27 Aug '18
Project: http://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-fc28-x8…
Build: http://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-fc28-x8…
Build Number: 40
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #40
[Yuval Turgeman] automation: remove build-artifacts* from stdci-v2
-----------------
Failed Tests:
-----------------
No tests ran.
1
1
Build failed in Jenkins:
system-sync_mirrors-centos-updates-el7-x86_64 #1880
by jenkins＠jenkins.phx.ovirt.org 26 Aug '18
by jenkins＠jenkins.phx.ovirt.org 26 Aug '18
26 Aug '18
See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…>
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…>
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 3afc9ba57755a1b2574f7a177184eb66d5f6a90d (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 3afc9ba57755a1b2574f7a177184eb66d5f6a90d
Commit message: "ovirt-release: drop non working jobs"
> git rev-list --no-walk 3afc9ba57755a1b2574f7a177184eb66d5f6a90d # timeout=10
[system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins5224362443311299032.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf
Checking if mirror needs a resync
Traceback (most recent call last):
File "/usr/bin/reposync", line 343, in <module>
main()
File "/usr/bin/reposync", line 175, in main
my.doRepoSetup()
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
return self._getRepos(thisrepo, True)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
self._repos.doSetup(thisrepo)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
dl = repo._async and repo._commonLoadRepoXML(repo)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML
local = self.cachedir + '/repomd.xml'
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda>
cachedir = property(lambda self: self._dirGetAttr('cachedir'))
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr
self.dirSetup()
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup
self._dirSetupMkdir_p(dir)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p
raise Errors.RepoError, msg
yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7'
Build step 'Execute shell' marked build as failure
1
3
[oVirt Jenkins] ovirt-system-tests_he-basic-suite-4.2 - Build # 510
- Failure!
by jenkins＠jenkins.phx.ovirt.org 26 Aug '18
by jenkins＠jenkins.phx.ovirt.org 26 Aug '18
26 Aug '18
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.2/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.2/510/
Build Number: 510
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #510
[Roy Golan] Introduce openshift-on-ovirt
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: 006_network_by_label.assign_hosts_network_label
Error Message:
status: 409
reason: Conflict
detail: Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later.
-------------------- >> begin captured logging << --------------------
lago.utils: ERROR: Error while running thread
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue
queue.put({'return': func()})
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/006_network_by_label.py", line 57, in _assign_host_network_label
host_nic=nic
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 16231, in add
headers={"Correlation-Id":correlation_id, "Expect":expect}
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add
return self.request('POST', url, body, headers, cls=cls)
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request
persistent_auth=self.__persistent_auth
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request
persistent_auth)
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request
raise errors.RequestError(response_code, response_reason, response_body)
RequestError:
status: 409
reason: Conflict
detail: Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later.
--------------------- >> end captured logging << ---------------------
Stack Trace:
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 69, in wrapper
return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs)
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/006_network_by_label.py", line 65, in assign_hosts_network_label
nt.assert_true(all(vt.join_all()))
File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue
queue.put({'return': func()})
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/006_network_by_label.py", line 57, in _assign_host_network_label
host_nic=nic
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 16231, in add
headers={"Correlation-Id":correlation_id, "Expect":expect}
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add
return self.request('POST', url, body, headers, cls=cls)
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request
persistent_auth=self.__persistent_auth
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request
persistent_auth)
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request
raise errors.RequestError(response_code, response_reason, response_body)
'\r\nstatus: 409\r\nreason: Conflict\r\ndetail: Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later.\n-------------------- >> begin captured logging << --------------------\nlago.utils: ERROR: Error while running thread\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue\n queue.put({\'return\': func()})\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/006_network_by_label.py", line 57, in _assign_host_network_label\n host_nic=nic\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 16231, in add\n headers={"Correlation-Id":correlation_id, "Expect":expect}\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add\n return self.request(\'POST\', url, body, headers, cls=cls)\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request\n persistent_auth=self.__persistent_auth\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request\n persistent_auth)\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request\n raise errors.RequestError(response_code, response_reason, response_body)\nRequestError: \r\nstatus: 409\r\nreason: Conflict\r\ndetail: Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later.\n--------------------- >> end captured logging << ---------------------'
1
2