- Infra - oVirt List Archives

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-4.2 - Build # 286 - Failure!
by jenkins＠jenkins.phx.ovirt.org 23 Jun '18

23 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/286/ Build Number: 286 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #286 [Sandro Bonazzola] yum-repos: master: adjusted for new packages ----------------- Failed Tests: ----------------- No tests ran.

1 2

[oVirt Jenkins] ovirt-system-tests_he-basic-suite-4.2 - Build # 316 - Failure!
by jenkins＠jenkins.phx.ovirt.org 23 Jun '18

23 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.2/316/ Build Number: 316 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #316 [Sandro Bonazzola] yum-repos: master: adjusted for new packages ----------------- Failed Tests: ----------------- All tests passed

1 1

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build # 506 - Failure!
by jenkins＠jenkins.phx.ovirt.org 23 Jun '18

23 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/506/ Build Number: 506 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #506 [Sandro Bonazzola] yum-repos: master: adjusted for new packages ----------------- Failed Tests: ----------------- No tests ran.

1 1

[oVirt Jenkins] ovirt-node-ng_ovirt-4.2-pre_build-artifacts-el7-x86_64 - Build # 391 - Failure!
by jenkins＠jenkins.phx.ovirt.org 23 Jun '18

23 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-4.2-pre_build-artifacts-el… Build: http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-4.2-pre_build-artifacts-el… Build Number: 391 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #391 [Francesco Romani] whitelist: add Tomasz Barański [Ryan Barry] kickstart: let VDSM reconfigure itself ----------------- Failed Tests: ----------------- No tests ran.

1 1

Build failed in Jenkins: system-sync_mirrors-epel-el6-x86_64 #1607
by jenkins＠jenkins.phx.ovirt.org 23 Jun '18

23 Jun '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/1607/displ…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision ff4050330abc014f5d54d61198050d4b9dc5db3a (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f ff4050330abc014f5d54d61198050d4b9dc5db3a Commit message: "Move repoman project to STDCIv2" > git rev-list --no-walk ff4050330abc014f5d54d61198050d4b9dc5db3a # timeout=10 [system-sync_mirrors-epel-el6-x86_64] $ /bin/bash -xe /tmp/jenkins7207805546833117692.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror epel-el6 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1474, in _commonLoadRepoXML if self._latestRepoXML(local): File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1452, in _latestRepoXML repomd = self.metalink_data.repomd File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 918, in <lambda> metalink_data = property(fget=lambda self: self._getMetalink(), File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 906, in _getMetalink shutil.move(result, self.metalink_filename) File "/usr/lib64/python2.7/shutil.py", line 301, in move copy2(src, real_dst) File "/usr/lib64/python2.7/shutil.py", line 130, in copy2 copyfile(src, dst) File "/usr/lib64/python2.7/shutil.py", line 82, in copyfile with open(src, 'rb') as fsrc: IOError: [Errno 2] No such file or directory: '/home/jenkins/mirrors_cache/fedora-base-fc25/metalink.xml.tmp' Build step 'Execute shell' marked build as failure

1 1

[oVirt Jenkins] ovirt-system-tests_he-basic-ansible-suite-4.2 - Build # 310 - Failure!
by jenkins＠jenkins.phx.ovirt.org 23 Jun '18

23 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4.2/… Build Number: 310 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #310 [Sandro Bonazzola] yum-repos: master: adjusted for new packages ----------------- Failed Tests: ----------------- All tests passed

1 1

[oVirt Jenkins] ovirt-node-ng_4.2_build-artifacts-el7-x86_64 - Build # 111 - Failure!
by jenkins＠jenkins.phx.ovirt.org 23 Jun '18

23 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-node-ng_4.2_build-artifacts-el7-x86_64/ Build: http://jenkins.ovirt.org/job/ovirt-node-ng_4.2_build-artifacts-el7-x86_64/1… Build Number: 111 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #111 [Daniel Belenky] Install python-six during global-setup [Barak Korren] Make mock_runner use curl instead of wget [Barak Korren] Fix proxied configuration for FC27 and FC28 [Daniel Belenky] Move project functions to a shared lib [Daniel Belenky] Utilize stdci_project groovy lib in CQ [Daniel Belenky] Move get_generic_queue_build_args to stdci_project [Ryan Barry] kickstart: let VDSM reconfigure itself ----------------- Failed Tests: ----------------- No tests ran.

1 2

[oVirt Jenkins] ovirt-system-tests_he-basic-ansible-suite-master - Build # 380 - Failure!
by jenkins＠jenkins.phx.ovirt.org 23 Jun '18

23 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-mast… Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-mast… Build Number: 380 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #380 [Sandro Bonazzola] yum-repos: master: adjusted for new packages ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 008_restart_he_vm.restart_he_vm Error Message: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2361 (Fri Jun 22 18:49:00 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2361 (Fri Jun 22 18:49:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "39ce70cf", "local_conf_timestamp": 2361, "host-ts": 2361}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2360 (Fri Jun 22 18:49:00 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2360 (Fri Jun 22 18:49:00 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host- -------------------- >> begin captured logging << -------------------- lago.ssh: DEBUG: start task:0206c430-f39c-48ba-ae10-463b259c7695:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:0206c430-f39c-48ba-ae10-463b259c7695:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running fc116bfe on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command fc116bfe on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command fc116bfe on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2152 (Fri Jun 22 18:45:31 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2152 (Fri Jun 22 18:45:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ac489e25", "local_conf_timestamp": 2152, "host-ts": 2152}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2150 (Fri Jun 22 18:45:30 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2151 (Fri Jun 22 18:45:31 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c5048dc4", "local_conf_timestamp": 2151, "host-ts": 2150}, "global_maintenance": true} root: INFO: * Shutting down HE VM on host: lago-he-basic-ansible-suite-master-host-0 lago.ssh: DEBUG: start task:309e4d1e-33d8-40b9-b815-629f19b32c19:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:309e4d1e-33d8-40b9-b815-629f19b32c19:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running fc941d6a on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-shutdown lago.ssh: DEBUG: Command fc941d6a on lago-he-basic-ansible-suite-master-host-0 returned with 0 root: INFO: * Command succeeded root: INFO: * Waiting for VM to be down... lago.ssh: DEBUG: start task:5c70f09d-db34-4dd1-9bd3-fffa981a46cd:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:5c70f09d-db34-4dd1-9bd3-fffa981a46cd:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running fd13e4b4 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command fd13e4b4 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command fd13e4b4 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2152 (Fri Jun 22 18:45:31 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2152 (Fri Jun 22 18:45:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ac489e25", "local_conf_timestamp": 2152, "host-ts": 2152}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2150 (Fri Jun 22 18:45:30 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2151 (Fri Jun 22 18:45:31 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c5048dc4", "local_conf_timestamp": 2151, "host-ts": 2150}, "global_maintenance": true} lago.ssh: DEBUG: start task:837c46d4-7157-46ad-b124-6a7a75723ad5:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:837c46d4-7157-46ad-b124-6a7a75723ad5:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 03b0c35a on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 03b0c35a on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 03b0c35a on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2162 (Fri Jun 22 18:45:42 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2163 (Fri Jun 22 18:45:42 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "6119f9bf", "local_conf_timestamp": 2163, "host-ts": 2162}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2160 (Fri Jun 22 18:45:40 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2160 (Fri Jun 22 18:45:40 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "aa074cd5", "local_conf_timestamp": 2160, "host-ts": 2160}, "global_maintenance": true} lago.ssh: DEBUG: start task:c1739841-aa58-424f-811c-9bb089e719c7:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:c1739841-aa58-424f-811c-9bb089e719c7:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 06090d38 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 06090d38 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 06090d38 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2172 (Fri Jun 22 18:45:52 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2172 (Fri Jun 22 18:45:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "3b8d448e", "local_conf_timestamp": 2172, "host-ts": 2172}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2170 (Fri Jun 22 18:45:50 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2171 (Fri Jun 22 18:45:51 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d503d33e", "local_conf_timestamp": 2171, "host-ts": 2170}, "global_maintenance": true} lago.ssh: DEBUG: start task:07646a89-9b6c-49dd-8346-1149320f2574:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:07646a89-9b6c-49dd-8346-1149320f2574:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 085ce974 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 085ce974 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 085ce974 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2172 (Fri Jun 22 18:45:52 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2172 (Fri Jun 22 18:45:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "3b8d448e", "local_conf_timestamp": 2172, "host-ts": 2172}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2170 (Fri Jun 22 18:45:50 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2171 (Fri Jun 22 18:45:51 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d503d33e", "local_conf_timestamp": 2171, "host-ts": 2170}, "global_maintenance": true} lago.ssh: DEBUG: start task:ec34da5e-75fa-4c2d-a31c-10b0557f7aae:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:ec34da5e-75fa-4c2d-a31c-10b0557f7aae:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 0aaa42c6 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 0aaa42c6 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 0aaa42c6 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2182 (Fri Jun 22 18:46:02 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2183 (Fri Jun 22 18:46:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e2fccec7", "local_conf_timestamp": 2183, "host-ts": 2182}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2180 (Fri Jun 22 18:46:00 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2180 (Fri Jun 22 18:46:00 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "da13d433", "local_conf_timestamp": 2180, "host-ts": 2180}, "global_maintenance": true} root: INFO: * VM is down. root: INFO: * Restarting services... lago.ssh: DEBUG: start task:0c52e610-8de0-473d-bab5-9b05bf3e3f38:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:0c52e610-8de0-473d-bab5-9b05bf3e3f38:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 0b29acdc on lago-he-basic-ansible-suite-master-host-0: systemctl restart vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command 0b29acdc on lago-he-basic-ansible-suite-master-host-0 returned with 0 root: INFO: * Success. root: INFO: * Waiting for agent to be ready... lago.ssh: DEBUG: start task:90ca3da8-f107-44cf-b5bd-282eeffde720:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:90ca3da8-f107-44cf-b5bd-282eeffde720:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 15923248 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 15923248 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 15923248 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:1b0a1bcd-1ef1-4e8c-be38-87e494eb9932:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:1b0a1bcd-1ef1-4e8c-be38-87e494eb9932:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 1bf07e92 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 1bf07e92 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 1bf07e92 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:bd26a276-9760-4946-9edc-0f8e62b4bc04:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:bd26a276-9760-4946-9edc-0f8e62b4bc04:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 1e242678 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 1e242678 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 1e242678 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:e283c4b9-724a-4e16-8410-5a31750a7d7b:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:e283c4b9-724a-4e16-8410-5a31750a7d7b:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 2052e6b4 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 2052e6b4 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 2052e6b4 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:b3e4245b-c3d1-4b90-aada-59901a1d9455:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:b3e4245b-c3d1-4b90-aada-59901a1d9455:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 22807ff0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 22807ff0 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 22807ff0 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:d6d3ee7e-86bc-4225-b2e7-e5fb3c26b36e:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:d6d3ee7e-86bc-4225-b2e7-e5fb3c26b36e:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 24b0c154 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 24b0c154 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 24b0c154 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:65126cb6-38ed-4953-9355-e0a74e23c8d5:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:65126cb6-38ed-4953-9355-e0a74e23c8d5:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 26e24894 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 26e24894 on lago-he-basic-ansible-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 26e24894 on lago-he-basic-ansible-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:61a3b3f4-8ff3-4c72-97b9-76451337edda:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:61a3b3f4-8ff3-4c72-97b9-76451337edda:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 29211734 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 29211734 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 29211734 on lago-he-basic-ansible-suite-master-host-0 output: !! Cluster is in GLOBAL MAINTENANCE mode !! --== Host lago-he-basic-ansible-suite-master-host-0 (id: 1) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : lago-he-basic-ansible-suite-master-host-0 Host ID : 1 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"} Score : 0 stopped : False Local maintenance : False crc32 : 2b063a5e local_conf_timestamp : 2231 Host timestamp : 2231 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2231 (Fri Jun 22 18:46:50 2018) host-id=1 score=0 vm_conf_refresh_time=2231 (Fri Jun 22 18:46:50 2018) conf_on_shared_storage=True maintenance=False state=ReinitializeFSM stopped=False --== Host lago-he-basic-ansible-suite-master-host-1 (id: 2) status ==-- conf_on_shared_storage : True Status up-to-date : False Hostname : lago-he-basic-ansible-suite-master-host-1 Host ID : 2 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : f1ce4c2e local_conf_timestamp : 2230 Host timestamp : 2230 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2230 (Fri Jun 22 18:46:50 2018) host-id=2 score=3400 vm_conf_refresh_time=2230 (Fri Jun 22 18:46:50 2018) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False !! Cluster is in GLOBAL MAINTENANCE mode !! root: INFO: * Agent is ready. root: INFO: * Starting VM... lago.ssh: DEBUG: start task:33e28fb6-a232-4363-a3a1-19f7fcd59851:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:33e28fb6-a232-4363-a3a1-19f7fcd59851:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 29a01732 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-start lago.ssh: DEBUG: Command 29a01732 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 29a01732 on lago-he-basic-ansible-suite-master-host-0 output: VM exists and is down, cleaning up and restarting root: INFO: * Command succeeded root: INFO: * Waiting to VM to be UP... lago.ssh: DEBUG: start task:c2293362-1b0f-4b35-85ed-0e9beb375b6f:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:c2293362-1b0f-4b35-85ed-0e9beb375b6f:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 2b02fc8e on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 2b02fc8e on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 2b02fc8e on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2231 (Fri Jun 22 18:46:50 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=2231 (Fri Jun 22 18:46:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=ReinitializeFSM\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "2b063a5e", "local_conf_timestamp": 2231, "host-ts": 2231}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2230 (Fri Jun 22 18:46:50 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2230 (Fri Jun 22 18:46:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f1ce4c2e", "local_conf_timestamp": 2230, "host-ts": 2230}, "global_maintenance": true} lago.ssh: DEBUG: start task:30716cb3-bc59-4fc0-b300-af571804841a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:30716cb3-bc59-4fc0-b300-af571804841a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 318195b6 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 318195b6 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 318195b6 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2241 (Fri Jun 22 18:47:01 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2241 (Fri Jun 22 18:47:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "abd1bf0a", "local_conf_timestamp": 2241, "host-ts": 2241}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2240 (Fri Jun 22 18:47:00 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2241 (Fri Jun 22 18:47:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "bec330cb", "local_conf_timestamp": 2241, "host-ts": 2240}, "global_maintenance": true} root: INFO: * VM is UP. root: INFO: * Waiting for engine to start ... lago.ssh: DEBUG: start task:f7581d83-a119-4ce7-af34-b6421741d98c:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:f7581d83-a119-4ce7-af34-b6421741d98c:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 31f8ab92 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 31f8ab92 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 31f8ab92 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2241 (Fri Jun 22 18:47:01 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2241 (Fri Jun 22 18:47:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "abd1bf0a", "local_conf_timestamp": 2241, "host-ts": 2241}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2240 (Fri Jun 22 18:47:00 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2241 (Fri Jun 22 18:47:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "bec330cb", "local_conf_timestamp": 2241, "host-ts": 2240}, "global_maintenance": true} lago.ssh: DEBUG: start task:66d9af16-ff39-4eb6-947b-36bb82f6c3c6:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:66d9af16-ff39-4eb6-947b-36bb82f6c3c6:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 38864f28 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 38864f28 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 38864f28 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2251 (Fri Jun 22 18:47:11 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2252 (Fri Jun 22 18:47:11 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "97357a3e", "local_conf_timestamp": 2252, "host-ts": 2251}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2250 (Fri Jun 22 18:47:10 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2250 (Fri Jun 22 18:47:10 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c1c7af20", "local_conf_timestamp": 2250, "host-ts": 2250}, "global_maintenance": true} lago.ssh: DEBUG: start task:d3b6a09a-2398-469b-a9e0-7c29eb2c7ee3:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:d3b6a09a-2398-469b-a9e0-7c29eb2c7ee3:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 3adfd514 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 3adfd514 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 3adfd514 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2261 (Fri Jun 22 18:47:20 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2261 (Fri Jun 22 18:47:20 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "3b7b9057", "local_conf_timestamp": 2261, "host-ts": 2261}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2250 (Fri Jun 22 18:47:10 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2250 (Fri Jun 22 18:47:10 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c1c7af20", "local_conf_timestamp": 2250, "host-ts": 2250}, "global_maintenance": true} lago.ssh: DEBUG: start task:38017336-63a8-487e-b3cf-b8770ec8df6b:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:38017336-63a8-487e-b3cf-b8770ec8df6b:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 3d2b870a on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 3d2b870a on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 3d2b870a on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2261 (Fri Jun 22 18:47:20 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2261 (Fri Jun 22 18:47:20 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "3b7b9057", "local_conf_timestamp": 2261, "host-ts": 2261}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2250 (Fri Jun 22 18:47:10 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2250 (Fri Jun 22 18:47:10 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c1c7af20", "local_conf_timestamp": 2250, "host-ts": 2250}, "global_maintenance": true} lago.ssh: DEBUG: start task:6e2a1c23-fab1-4778-b158-6c27bd95bdb2:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:6e2a1c23-fab1-4778-b158-6c27bd95bdb2:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 3f871f1e on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 3f871f1e on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 3f871f1e on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2271 (Fri Jun 22 18:47:31 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2271 (Fri Jun 22 18:47:31 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9e9604d9", "local_conf_timestamp": 2271, "host-ts": 2271}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2270 (Fri Jun 22 18:47:30 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2270 (Fri Jun 22 18:47:30 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d1c0f1da", "local_conf_timestamp": 2270, "host-ts": 2270}, "global_maintenance": true} lago.ssh: DEBUG: start task:aeb7803a-884e-4cf7-abae-50dc4a196ad3:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:aeb7803a-884e-4cf7-abae-50dc4a196ad3:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 41ef5de8 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 41ef5de8 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 41ef5de8 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2271 (Fri Jun 22 18:47:31 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2271 (Fri Jun 22 18:47:31 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9e9604d9", "local_conf_timestamp": 2271, "host-ts": 2271}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2270 (Fri Jun 22 18:47:30 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2270 (Fri Jun 22 18:47:30 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d1c0f1da", "local_conf_timestamp": 2270, "host-ts": 2270}, "global_maintenance": true} lago.ssh: DEBUG: start task:6fcec395-edba-49e7-af0b-905507b437b4:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:6fcec395-edba-49e7-af0b-905507b437b4:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 44413b34 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 44413b34 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 44413b34 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2271 (Fri Jun 22 18:47:31 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2271 (Fri Jun 22 18:47:31 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9e9604d9", "local_conf_timestamp": 2271, "host-ts": 2271}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2270 (Fri Jun 22 18:47:30 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2270 (Fri Jun 22 18:47:30 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d1c0f1da", "local_conf_timestamp": 2270, "host-ts": 2270}, "global_maintenance": true} lago.ssh: DEBUG: start task:6ad23fd0-beaa-4dad-a947-2a56ff4f332b:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:6ad23fd0-beaa-4dad-a947-2a56ff4f332b:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 4705f74c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 4705f74c on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 4705f74c on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2281 (Fri Jun 22 18:47:40 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2281 (Fri Jun 22 18:47:40 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7ecf5046", "local_conf_timestamp": 2281, "host-ts": 2281}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2270 (Fri Jun 22 18:47:30 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2270 (Fri Jun 22 18:47:30 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d1c0f1da", "local_conf_timestamp": 2270, "host-ts": 2270}, "global_maintenance": true} lago.ssh: DEBUG: start task:79f6e898-bd3a-4f47-8c6c-73d89befd8ac:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:79f6e898-bd3a-4f47-8c6c-73d89befd8ac:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 49860688 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 49860688 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 49860688 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2281 (Fri Jun 22 18:47:40 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2281 (Fri Jun 22 18:47:40 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7ecf5046", "local_conf_timestamp": 2281, "host-ts": 2281}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2270 (Fri Jun 22 18:47:30 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2270 (Fri Jun 22 18:47:30 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d1c0f1da", "local_conf_timestamp": 2270, "host-ts": 2270}, "global_maintenance": true} lago.ssh: DEBUG: start task:180e1799-a99a-4168-9d31-1117997ea17a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:180e1799-a99a-4168-9d31-1117997ea17a:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 4c0309c4 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 4c0309c4 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 4c0309c4 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2291 (Fri Jun 22 18:47:50 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2291 (Fri Jun 22 18:47:51 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "3e46b69b", "local_conf_timestamp": 2291, "host-ts": 2291}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2290 (Fri Jun 22 18:47:50 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2290 (Fri Jun 22 18:47:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a1d4693c", "local_conf_timestamp": 2290, "host-ts": 2290}, "global_maintenance": true} lago.ssh: DEBUG: start task:0c42481f-ceb3-490a-b03d-eabf81706238:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:0c42481f-ceb3-490a-b03d-eabf81706238:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 4e78e246 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 4e78e246 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 4e78e246 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2291 (Fri Jun 22 18:47:50 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2291 (Fri Jun 22 18:47:51 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "3e46b69b", "local_conf_timestamp": 2291, "host-ts": 2291}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2290 (Fri Jun 22 18:47:50 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2290 (Fri Jun 22 18:47:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a1d4693c", "local_conf_timestamp": 2290, "host-ts": 2290}, "global_maintenance": true} lago.ssh: DEBUG: start task:02efa199-5e91-40a7-b99c-c21af182fc39:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:02efa199-5e91-40a7-b99c-c21af182fc39:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 50d382e4 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 50d382e4 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 50d382e4 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2301 (Fri Jun 22 18:48:00 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2301 (Fri Jun 22 18:48:00 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "54c62251", "local_conf_timestamp": 2301, "host-ts": 2301}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2300 (Fri Jun 22 18:48:00 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2300 (Fri Jun 22 18:48:00 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c776b387", "local_conf_timestamp": 2300, "host-ts": 2300}, "global_maintenance": true} lago.ssh: DEBUG: start task:c59c7606-8685-402a-a5c2-6279e0376859:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:c59c7606-8685-402a-a5c2-6279e0376859:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 531da2dc on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 531da2dc on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 531da2dc on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2301 (Fri Jun 22 18:48:00 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2301 (Fri Jun 22 18:48:00 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "54c62251", "local_conf_timestamp": 2301, "host-ts": 2301}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2300 (Fri Jun 22 18:48:00 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2300 (Fri Jun 22 18:48:00 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c776b387", "local_conf_timestamp": 2300, "host-ts": 2300}, "global_maintenance": true} lago.ssh: DEBUG: start task:a41676e7-26b6-4f4b-ac81-fd2c7eac215d:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:a41676e7-26b6-4f4b-ac81-fd2c7eac215d:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 557f2690 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 557f2690 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 557f2690 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2301 (Fri Jun 22 18:48:00 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2301 (Fri Jun 22 18:48:00 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "54c62251", "local_conf_timestamp": 2301, "host-ts": 2301}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2300 (Fri Jun 22 18:48:00 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2300 (Fri Jun 22 18:48:00 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c776b387", "local_conf_timestamp": 2300, "host-ts": 2300}, "global_maintenance": true} lago.ssh: DEBUG: start task:4db92d8b-0849-462b-86aa-56aa1fadbdc8:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:4db92d8b-0849-462b-86aa-56aa1fadbdc8:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 581c0b02 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 581c0b02 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 581c0b02 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2311 (Fri Jun 22 18:48:11 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2311 (Fri Jun 22 18:48:11 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "46ba3a94", "local_conf_timestamp": 2311, "host-ts": 2311}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2310 (Fri Jun 22 18:48:10 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2311 (Fri Jun 22 18:48:11 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b8722c6c", "local_conf_timestamp": 2311, "host-ts": 2310}, "global_maintenance": true} lago.ssh: DEBUG: start task:3bab9298-81ac-4629-8ca0-e498247bfbc7:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:3bab9298-81ac-4629-8ca0-e498247bfbc7:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 5a915d42 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 5a915d42 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 5a915d42 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2311 (Fri Jun 22 18:48:11 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2311 (Fri Jun 22 18:48:11 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "46ba3a94", "local_conf_timestamp": 2311, "host-ts": 2311}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2310 (Fri Jun 22 18:48:10 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2311 (Fri Jun 22 18:48:11 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b8722c6c", "local_conf_timestamp": 2311, "host-ts": 2310}, "global_maintenance": true} lago.ssh: DEBUG: start task:6b9937a8-8904-491a-ac42-1c0939abbc8f:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:6b9937a8-8904-491a-ac42-1c0939abbc8f:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 5d137744 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 5d137744 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 5d137744 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2321 (Fri Jun 22 18:48:20 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2321 (Fri Jun 22 18:48:20 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "703e13db", "local_conf_timestamp": 2321, "host-ts": 2321}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2320 (Fri Jun 22 18:48:20 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2320 (Fri Jun 22 18:48:20 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d771ed7d", "local_conf_timestamp": 2320, "host-ts": 2320}, "global_maintenance": true} lago.ssh: DEBUG: start task:fc15aa76-0947-4ce4-9753-bfd35660dcd1:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:fc15aa76-0947-4ce4-9753-bfd35660dcd1:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 5f7ce786 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 5f7ce786 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 5f7ce786 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2321 (Fri Jun 22 18:48:20 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2321 (Fri Jun 22 18:48:20 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "703e13db", "local_conf_timestamp": 2321, "host-ts": 2321}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2320 (Fri Jun 22 18:48:20 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2320 (Fri Jun 22 18:48:20 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d771ed7d", "local_conf_timestamp": 2320, "host-ts": 2320}, "global_maintenance": true} lago.ssh: DEBUG: start task:83dc71d9-013e-4c66-a2d1-d12cc9adddd0:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:83dc71d9-013e-4c66-a2d1-d12cc9adddd0:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 61eef7c0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 61eef7c0 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 61eef7c0 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2321 (Fri Jun 22 18:48:20 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2321 (Fri Jun 22 18:48:20 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "703e13db", "local_conf_timestamp": 2321, "host-ts": 2321}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2320 (Fri Jun 22 18:48:20 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2320 (Fri Jun 22 18:48:20 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d771ed7d", "local_conf_timestamp": 2320, "host-ts": 2320}, "global_maintenance": true} lago.ssh: DEBUG: start task:0c3f4987-1d01-46dc-aa66-c3d0c6933430:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:0c3f4987-1d01-46dc-aa66-c3d0c6933430:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 64b46030 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 64b46030 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 64b46030 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2331 (Fri Jun 22 18:48:31 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2331 (Fri Jun 22 18:48:31 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "62420b1e", "local_conf_timestamp": 2331, "host-ts": 2331}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2330 (Fri Jun 22 18:48:30 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2331 (Fri Jun 22 18:48:31 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a8757296", "local_conf_timestamp": 2331, "host-ts": 2330}, "global_maintenance": true} lago.ssh: DEBUG: start task:baa23295-8069-4395-8756-bc5db853ff9d:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:baa23295-8069-4395-8756-bc5db853ff9d:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 6720458c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 6720458c on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 6720458c on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2331 (Fri Jun 22 18:48:31 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2331 (Fri Jun 22 18:48:31 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "62420b1e", "local_conf_timestamp": 2331, "host-ts": 2331}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2330 (Fri Jun 22 18:48:30 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2331 (Fri Jun 22 18:48:31 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a8757296", "local_conf_timestamp": 2331, "host-ts": 2330}, "global_maintenance": true} lago.ssh: DEBUG: start task:479ef1da-24e7-490e-88ed-c61739a044d0:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:479ef1da-24e7-490e-88ed-c61739a044d0:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 699a90ec on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 699a90ec on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 699a90ec on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2341 (Fri Jun 22 18:48:40 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2341 (Fri Jun 22 18:48:40 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1d364145", "local_conf_timestamp": 2341, "host-ts": 2341}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2340 (Fri Jun 22 18:48:40 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2340 (Fri Jun 22 18:48:40 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e7780e73", "local_conf_timestamp": 2340, "host-ts": 2340}, "global_maintenance": true} lago.ssh: DEBUG: start task:24152678-5688-4c02-8b24-5f47ba7f2d3c:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:24152678-5688-4c02-8b24-5f47ba7f2d3c:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 6c2692ac on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 6c2692ac on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 6c2692ac on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2341 (Fri Jun 22 18:48:40 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2341 (Fri Jun 22 18:48:40 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1d364145", "local_conf_timestamp": 2341, "host-ts": 2341}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2340 (Fri Jun 22 18:48:40 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2340 (Fri Jun 22 18:48:40 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e7780e73", "local_conf_timestamp": 2340, "host-ts": 2340}, "global_maintenance": true} lago.ssh: DEBUG: start task:36dd77ba-9529-45b4-b449-b7722c0b98fe:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:36dd77ba-9529-45b4-b449-b7722c0b98fe:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 6ebb7244 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 6ebb7244 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 6ebb7244 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2341 (Fri Jun 22 18:48:40 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2341 (Fri Jun 22 18:48:40 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1d364145", "local_conf_timestamp": 2341, "host-ts": 2341}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2350 (Fri Jun 22 18:48:50 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2350 (Fri Jun 22 18:48:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ef7ba10e", "local_conf_timestamp": 2350, "host-ts": 2350}, "global_maintenance": true} lago.ssh: DEBUG: start task:257e02ab-bf9b-4ac2-8ff7-9515d4f1aa56:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:257e02ab-bf9b-4ac2-8ff7-9515d4f1aa56:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 7159a5e8 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 7159a5e8 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 7159a5e8 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2351 (Fri Jun 22 18:48:51 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2352 (Fri Jun 22 18:48:51 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "9643083a", "local_conf_timestamp": 2352, "host-ts": 2351}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2350 (Fri Jun 22 18:48:50 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2350 (Fri Jun 22 18:48:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ef7ba10e", "local_conf_timestamp": 2350, "host-ts": 2350}, "global_maintenance": true} lago.ssh: DEBUG: start task:27ec01d4-6761-4468-8ed4-6ab4fe4fcb00:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:27ec01d4-6761-4468-8ed4-6ab4fe4fcb00:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 73bd77e2 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 73bd77e2 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 73bd77e2 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2351 (Fri Jun 22 18:48:51 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2352 (Fri Jun 22 18:48:51 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "9643083a", "local_conf_timestamp": 2352, "host-ts": 2351}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2350 (Fri Jun 22 18:48:50 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2350 (Fri Jun 22 18:48:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ef7ba10e", "local_conf_timestamp": 2350, "host-ts": 2350}, "global_maintenance": true} lago.ssh: DEBUG: start task:cc1eb4e6-3fd2-47d2-8998-a0e9e91743f9:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:cc1eb4e6-3fd2-47d2-8998-a0e9e91743f9:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 76ff1456 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 76ff1456 on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 76ff1456 on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2361 (Fri Jun 22 18:49:00 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2361 (Fri Jun 22 18:49:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "39ce70cf", "local_conf_timestamp": 2361, "host-ts": 2361}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2360 (Fri Jun 22 18:49:00 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2360 (Fri Jun 22 18:49:00 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f77f5089", "local_conf_timestamp": 2360, "host-ts": 2360}, "global_maintenance": true} lago.ssh: DEBUG: start task:34564059-b09e-4c89-9a53-9a4a876e15d0:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: end task:34564059-b09e-4c89-9a53-9a4a876e15d0:Get ssh client for lago-he-basic-ansible-suite-master-host-0: lago.ssh: DEBUG: Running 796297fe on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 796297fe on lago-he-basic-ansible-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 796297fe on lago-he-basic-ansible-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2361 (Fri Jun 22 18:49:00 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2361 (Fri Jun 22 18:49:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "39ce70cf", "local_conf_timestamp": 2361, "host-ts": 2361}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2360 (Fri Jun 22 18:49:00 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2360 (Fri Jun 22 18:49:00 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host- ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f48cce38140> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 139, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 83, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) RuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2361 (Fri Jun 22 18:49:00 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2361 (Fri Jun 22 18:49:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "39ce70cf", "local_conf_timestamp": 2361, "host-ts": 2361}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2360 (Fri Jun 22 18:49:00 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2360 (Fri Jun 22 18:49:00 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host- --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 47, in restart_he_vm _wait_for_engine_health(host) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 137, in _wait_for_engine_health testlib.assert_true_within_long(lambda: any( File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 271, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 258, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 139, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 83, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) 'could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2361 (Fri Jun 22 18:49:00 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2361 (Fri Jun 22 18:49:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "39ce70cf", "local_conf_timestamp": 2361, "host-ts": 2361}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2360 (Fri Jun 22 18:49:00 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2360 (Fri Jun 22 18:49:00 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:0206c430-f39c-48ba-ae10-463b259c7695:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:0206c430-f39c-48ba-ae10-463b259c7695:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running fc116bfe on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fc116bfe on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command fc116bfe on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2152 (Fri Jun 22 18:45:31 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2152 (Fri Jun 22 18:45:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ac489e25", "local_conf_timestamp": 2152, "host-ts": 2152}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2150 (Fri Jun 22 18:45:30 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2151 (Fri Jun 22 18:45:31 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c5048dc4", "local_conf_timestamp": 2151, "host-ts": 2150}, "global_maintenance": true}\n\nroot: INFO: * Shutting down HE VM on host: lago-he-basic-ansible-suite-master-host-0\nlago.ssh: DEBUG: start task:309e4d1e-33d8-40b9-b815-629f19b32c19:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:309e4d1e-33d8-40b9-b815-629f19b32c19:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running fc941d6a on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-shutdown\nlago.ssh: DEBUG: Command fc941d6a on lago-he-basic-ansible-suite-master-host-0 returned with 0\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be down...\nlago.ssh: DEBUG: start task:5c70f09d-db34-4dd1-9bd3-fffa981a46cd:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:5c70f09d-db34-4dd1-9bd3-fffa981a46cd:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running fd13e4b4 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fd13e4b4 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command fd13e4b4 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2152 (Fri Jun 22 18:45:31 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2152 (Fri Jun 22 18:45:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ac489e25", "local_conf_timestamp": 2152, "host-ts": 2152}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2150 (Fri Jun 22 18:45:30 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2151 (Fri Jun 22 18:45:31 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c5048dc4", "local_conf_timestamp": 2151, "host-ts": 2150}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:837c46d4-7157-46ad-b124-6a7a75723ad5:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:837c46d4-7157-46ad-b124-6a7a75723ad5:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 03b0c35a on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 03b0c35a on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 03b0c35a on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2162 (Fri Jun 22 18:45:42 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2163 (Fri Jun 22 18:45:42 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "6119f9bf", "local_conf_timestamp": 2163, "host-ts": 2162}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2160 (Fri Jun 22 18:45:40 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2160 (Fri Jun 22 18:45:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "aa074cd5", "local_conf_timestamp": 2160, "host-ts": 2160}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:c1739841-aa58-424f-811c-9bb089e719c7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:c1739841-aa58-424f-811c-9bb089e719c7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 06090d38 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 06090d38 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 06090d38 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2172 (Fri Jun 22 18:45:52 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2172 (Fri Jun 22 18:45:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "3b8d448e", "local_conf_timestamp": 2172, "host-ts": 2172}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2170 (Fri Jun 22 18:45:50 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2171 (Fri Jun 22 18:45:51 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d503d33e", "local_conf_timestamp": 2171, "host-ts": 2170}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:07646a89-9b6c-49dd-8346-1149320f2574:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:07646a89-9b6c-49dd-8346-1149320f2574:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 085ce974 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 085ce974 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 085ce974 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2172 (Fri Jun 22 18:45:52 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2172 (Fri Jun 22 18:45:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "3b8d448e", "local_conf_timestamp": 2172, "host-ts": 2172}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2170 (Fri Jun 22 18:45:50 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2171 (Fri Jun 22 18:45:51 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d503d33e", "local_conf_timestamp": 2171, "host-ts": 2170}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ec34da5e-75fa-4c2d-a31c-10b0557f7aae:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:ec34da5e-75fa-4c2d-a31c-10b0557f7aae:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 0aaa42c6 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 0aaa42c6 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 0aaa42c6 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2182 (Fri Jun 22 18:46:02 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2183 (Fri Jun 22 18:46:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e2fccec7", "local_conf_timestamp": 2183, "host-ts": 2182}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2180 (Fri Jun 22 18:46:00 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2180 (Fri Jun 22 18:46:00 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "da13d433", "local_conf_timestamp": 2180, "host-ts": 2180}, "global_maintenance": true}\n\nroot: INFO: * VM is down.\nroot: INFO: * Restarting services...\nlago.ssh: DEBUG: start task:0c52e610-8de0-473d-bab5-9b05bf3e3f38:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:0c52e610-8de0-473d-bab5-9b05bf3e3f38:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 0b29acdc on lago-he-basic-ansible-suite-master-host-0: systemctl restart vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command 0b29acdc on lago-he-basic-ansible-suite-master-host-0 returned with 0\nroot: INFO: * Success.\nroot: INFO: * Waiting for agent to be ready...\nlago.ssh: DEBUG: start task:90ca3da8-f107-44cf-b5bd-282eeffde720:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:90ca3da8-f107-44cf-b5bd-282eeffde720:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 15923248 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 15923248 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 15923248 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:1b0a1bcd-1ef1-4e8c-be38-87e494eb9932:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:1b0a1bcd-1ef1-4e8c-be38-87e494eb9932:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 1bf07e92 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 1bf07e92 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 1bf07e92 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:bd26a276-9760-4946-9edc-0f8e62b4bc04:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:bd26a276-9760-4946-9edc-0f8e62b4bc04:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 1e242678 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 1e242678 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 1e242678 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:e283c4b9-724a-4e16-8410-5a31750a7d7b:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:e283c4b9-724a-4e16-8410-5a31750a7d7b:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 2052e6b4 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 2052e6b4 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 2052e6b4 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:b3e4245b-c3d1-4b90-aada-59901a1d9455:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:b3e4245b-c3d1-4b90-aada-59901a1d9455:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 22807ff0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 22807ff0 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 22807ff0 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:d6d3ee7e-86bc-4225-b2e7-e5fb3c26b36e:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:d6d3ee7e-86bc-4225-b2e7-e5fb3c26b36e:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 24b0c154 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 24b0c154 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 24b0c154 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:65126cb6-38ed-4953-9355-e0a74e23c8d5:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:65126cb6-38ed-4953-9355-e0a74e23c8d5:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 26e24894 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 26e24894 on lago-he-basic-ansible-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 26e24894 on lago-he-basic-ansible-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:61a3b3f4-8ff3-4c72-97b9-76451337edda:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:61a3b3f4-8ff3-4c72-97b9-76451337edda:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 29211734 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 29211734 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 29211734 on lago-he-basic-ansible-suite-master-host-0 output:\n \n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\n\n--== Host lago-he-basic-ansible-suite-master-host-0 (id: 1) status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : True\nHostname : lago-he-basic-ansible-suite-master-host-0\nHost ID : 1\nEngine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}\nScore : 0\nstopped : False\nLocal maintenance : False\ncrc32 : 2b063a5e\nlocal_conf_timestamp : 2231\nHost timestamp : 2231\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2231 (Fri Jun 22 18:46:50 2018)\n\thost-id=1\n\tscore=0\n\tvm_conf_refresh_time=2231 (Fri Jun 22 18:46:50 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=ReinitializeFSM\n\tstopped=False\n\n\n--== Host lago-he-basic-ansible-suite-master-host-1 (id: 2) status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : False\nHostname : lago-he-basic-ansible-suite-master-host-1\nHost ID : 2\nEngine status : unknown stale-data\nScore : 3400\nstopped : False\nLocal maintenance : False\ncrc32 : f1ce4c2e\nlocal_conf_timestamp : 2230\nHost timestamp : 2230\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2230 (Fri Jun 22 18:46:50 2018)\n\thost-id=2\n\tscore=3400\n\tvm_conf_refresh_time=2230 (Fri Jun 22 18:46:50 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=GlobalMaintenance\n\tstopped=False\n\n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\nroot: INFO: * Agent is ready.\nroot: INFO: * Starting VM...\nlago.ssh: DEBUG: start task:33e28fb6-a232-4363-a3a1-19f7fcd59851:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:33e28fb6-a232-4363-a3a1-19f7fcd59851:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 29a01732 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-start\nlago.ssh: DEBUG: Command 29a01732 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 29a01732 on lago-he-basic-ansible-suite-master-host-0 output:\n VM exists and is down, cleaning up and restarting\n\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting to VM to be UP...\nlago.ssh: DEBUG: start task:c2293362-1b0f-4b35-85ed-0e9beb375b6f:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:c2293362-1b0f-4b35-85ed-0e9beb375b6f:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 2b02fc8e on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 2b02fc8e on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 2b02fc8e on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2231 (Fri Jun 22 18:46:50 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=2231 (Fri Jun 22 18:46:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=ReinitializeFSM\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "2b063a5e", "local_conf_timestamp": 2231, "host-ts": 2231}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2230 (Fri Jun 22 18:46:50 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2230 (Fri Jun 22 18:46:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f1ce4c2e", "local_conf_timestamp": 2230, "host-ts": 2230}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:30716cb3-bc59-4fc0-b300-af571804841a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:30716cb3-bc59-4fc0-b300-af571804841a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 318195b6 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 318195b6 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 318195b6 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2241 (Fri Jun 22 18:47:01 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2241 (Fri Jun 22 18:47:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "abd1bf0a", "local_conf_timestamp": 2241, "host-ts": 2241}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2240 (Fri Jun 22 18:47:00 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2241 (Fri Jun 22 18:47:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "bec330cb", "local_conf_timestamp": 2241, "host-ts": 2240}, "global_maintenance": true}\n\nroot: INFO: * VM is UP.\nroot: INFO: * Waiting for engine to start ...\nlago.ssh: DEBUG: start task:f7581d83-a119-4ce7-af34-b6421741d98c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:f7581d83-a119-4ce7-af34-b6421741d98c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 31f8ab92 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 31f8ab92 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 31f8ab92 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2241 (Fri Jun 22 18:47:01 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2241 (Fri Jun 22 18:47:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "abd1bf0a", "local_conf_timestamp": 2241, "host-ts": 2241}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2240 (Fri Jun 22 18:47:00 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2241 (Fri Jun 22 18:47:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "bec330cb", "local_conf_timestamp": 2241, "host-ts": 2240}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:66d9af16-ff39-4eb6-947b-36bb82f6c3c6:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:66d9af16-ff39-4eb6-947b-36bb82f6c3c6:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 38864f28 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 38864f28 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 38864f28 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2251 (Fri Jun 22 18:47:11 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2252 (Fri Jun 22 18:47:11 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "97357a3e", "local_conf_timestamp": 2252, "host-ts": 2251}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2250 (Fri Jun 22 18:47:10 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2250 (Fri Jun 22 18:47:10 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c1c7af20", "local_conf_timestamp": 2250, "host-ts": 2250}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:d3b6a09a-2398-469b-a9e0-7c29eb2c7ee3:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:d3b6a09a-2398-469b-a9e0-7c29eb2c7ee3:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 3adfd514 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 3adfd514 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 3adfd514 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2261 (Fri Jun 22 18:47:20 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2261 (Fri Jun 22 18:47:20 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "3b7b9057", "local_conf_timestamp": 2261, "host-ts": 2261}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2250 (Fri Jun 22 18:47:10 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2250 (Fri Jun 22 18:47:10 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c1c7af20", "local_conf_timestamp": 2250, "host-ts": 2250}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:38017336-63a8-487e-b3cf-b8770ec8df6b:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:38017336-63a8-487e-b3cf-b8770ec8df6b:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 3d2b870a on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 3d2b870a on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 3d2b870a on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2261 (Fri Jun 22 18:47:20 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2261 (Fri Jun 22 18:47:20 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "3b7b9057", "local_conf_timestamp": 2261, "host-ts": 2261}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2250 (Fri Jun 22 18:47:10 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2250 (Fri Jun 22 18:47:10 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c1c7af20", "local_conf_timestamp": 2250, "host-ts": 2250}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:6e2a1c23-fab1-4778-b158-6c27bd95bdb2:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:6e2a1c23-fab1-4778-b158-6c27bd95bdb2:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 3f871f1e on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 3f871f1e on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 3f871f1e on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2271 (Fri Jun 22 18:47:31 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2271 (Fri Jun 22 18:47:31 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9e9604d9", "local_conf_timestamp": 2271, "host-ts": 2271}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2270 (Fri Jun 22 18:47:30 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2270 (Fri Jun 22 18:47:30 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d1c0f1da", "local_conf_timestamp": 2270, "host-ts": 2270}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:aeb7803a-884e-4cf7-abae-50dc4a196ad3:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:aeb7803a-884e-4cf7-abae-50dc4a196ad3:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 41ef5de8 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 41ef5de8 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 41ef5de8 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2271 (Fri Jun 22 18:47:31 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2271 (Fri Jun 22 18:47:31 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9e9604d9", "local_conf_timestamp": 2271, "host-ts": 2271}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2270 (Fri Jun 22 18:47:30 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2270 (Fri Jun 22 18:47:30 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d1c0f1da", "local_conf_timestamp": 2270, "host-ts": 2270}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:6fcec395-edba-49e7-af0b-905507b437b4:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:6fcec395-edba-49e7-af0b-905507b437b4:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 44413b34 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 44413b34 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 44413b34 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2271 (Fri Jun 22 18:47:31 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2271 (Fri Jun 22 18:47:31 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9e9604d9", "local_conf_timestamp": 2271, "host-ts": 2271}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2270 (Fri Jun 22 18:47:30 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2270 (Fri Jun 22 18:47:30 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d1c0f1da", "local_conf_timestamp": 2270, "host-ts": 2270}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:6ad23fd0-beaa-4dad-a947-2a56ff4f332b:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:6ad23fd0-beaa-4dad-a947-2a56ff4f332b:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 4705f74c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 4705f74c on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 4705f74c on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2281 (Fri Jun 22 18:47:40 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2281 (Fri Jun 22 18:47:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7ecf5046", "local_conf_timestamp": 2281, "host-ts": 2281}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2270 (Fri Jun 22 18:47:30 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2270 (Fri Jun 22 18:47:30 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d1c0f1da", "local_conf_timestamp": 2270, "host-ts": 2270}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:79f6e898-bd3a-4f47-8c6c-73d89befd8ac:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:79f6e898-bd3a-4f47-8c6c-73d89befd8ac:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 49860688 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 49860688 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 49860688 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2281 (Fri Jun 22 18:47:40 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2281 (Fri Jun 22 18:47:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7ecf5046", "local_conf_timestamp": 2281, "host-ts": 2281}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2270 (Fri Jun 22 18:47:30 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2270 (Fri Jun 22 18:47:30 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d1c0f1da", "local_conf_timestamp": 2270, "host-ts": 2270}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:180e1799-a99a-4168-9d31-1117997ea17a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:180e1799-a99a-4168-9d31-1117997ea17a:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 4c0309c4 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 4c0309c4 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 4c0309c4 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2291 (Fri Jun 22 18:47:50 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2291 (Fri Jun 22 18:47:51 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "3e46b69b", "local_conf_timestamp": 2291, "host-ts": 2291}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2290 (Fri Jun 22 18:47:50 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2290 (Fri Jun 22 18:47:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a1d4693c", "local_conf_timestamp": 2290, "host-ts": 2290}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:0c42481f-ceb3-490a-b03d-eabf81706238:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:0c42481f-ceb3-490a-b03d-eabf81706238:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 4e78e246 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 4e78e246 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 4e78e246 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2291 (Fri Jun 22 18:47:50 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2291 (Fri Jun 22 18:47:51 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "3e46b69b", "local_conf_timestamp": 2291, "host-ts": 2291}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2290 (Fri Jun 22 18:47:50 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2290 (Fri Jun 22 18:47:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a1d4693c", "local_conf_timestamp": 2290, "host-ts": 2290}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:02efa199-5e91-40a7-b99c-c21af182fc39:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:02efa199-5e91-40a7-b99c-c21af182fc39:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 50d382e4 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 50d382e4 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 50d382e4 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2301 (Fri Jun 22 18:48:00 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2301 (Fri Jun 22 18:48:00 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "54c62251", "local_conf_timestamp": 2301, "host-ts": 2301}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2300 (Fri Jun 22 18:48:00 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2300 (Fri Jun 22 18:48:00 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c776b387", "local_conf_timestamp": 2300, "host-ts": 2300}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:c59c7606-8685-402a-a5c2-6279e0376859:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:c59c7606-8685-402a-a5c2-6279e0376859:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 531da2dc on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 531da2dc on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 531da2dc on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2301 (Fri Jun 22 18:48:00 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2301 (Fri Jun 22 18:48:00 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "54c62251", "local_conf_timestamp": 2301, "host-ts": 2301}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2300 (Fri Jun 22 18:48:00 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2300 (Fri Jun 22 18:48:00 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c776b387", "local_conf_timestamp": 2300, "host-ts": 2300}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:a41676e7-26b6-4f4b-ac81-fd2c7eac215d:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:a41676e7-26b6-4f4b-ac81-fd2c7eac215d:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 557f2690 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 557f2690 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 557f2690 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2301 (Fri Jun 22 18:48:00 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2301 (Fri Jun 22 18:48:00 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "54c62251", "local_conf_timestamp": 2301, "host-ts": 2301}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2300 (Fri Jun 22 18:48:00 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2300 (Fri Jun 22 18:48:00 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c776b387", "local_conf_timestamp": 2300, "host-ts": 2300}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:4db92d8b-0849-462b-86aa-56aa1fadbdc8:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:4db92d8b-0849-462b-86aa-56aa1fadbdc8:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 581c0b02 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 581c0b02 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 581c0b02 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2311 (Fri Jun 22 18:48:11 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2311 (Fri Jun 22 18:48:11 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "46ba3a94", "local_conf_timestamp": 2311, "host-ts": 2311}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2310 (Fri Jun 22 18:48:10 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2311 (Fri Jun 22 18:48:11 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b8722c6c", "local_conf_timestamp": 2311, "host-ts": 2310}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:3bab9298-81ac-4629-8ca0-e498247bfbc7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:3bab9298-81ac-4629-8ca0-e498247bfbc7:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 5a915d42 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 5a915d42 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 5a915d42 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2311 (Fri Jun 22 18:48:11 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2311 (Fri Jun 22 18:48:11 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "46ba3a94", "local_conf_timestamp": 2311, "host-ts": 2311}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2310 (Fri Jun 22 18:48:10 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2311 (Fri Jun 22 18:48:11 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b8722c6c", "local_conf_timestamp": 2311, "host-ts": 2310}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:6b9937a8-8904-491a-ac42-1c0939abbc8f:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:6b9937a8-8904-491a-ac42-1c0939abbc8f:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 5d137744 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 5d137744 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 5d137744 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2321 (Fri Jun 22 18:48:20 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2321 (Fri Jun 22 18:48:20 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "703e13db", "local_conf_timestamp": 2321, "host-ts": 2321}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2320 (Fri Jun 22 18:48:20 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2320 (Fri Jun 22 18:48:20 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d771ed7d", "local_conf_timestamp": 2320, "host-ts": 2320}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:fc15aa76-0947-4ce4-9753-bfd35660dcd1:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:fc15aa76-0947-4ce4-9753-bfd35660dcd1:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 5f7ce786 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 5f7ce786 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 5f7ce786 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2321 (Fri Jun 22 18:48:20 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2321 (Fri Jun 22 18:48:20 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "703e13db", "local_conf_timestamp": 2321, "host-ts": 2321}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2320 (Fri Jun 22 18:48:20 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2320 (Fri Jun 22 18:48:20 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d771ed7d", "local_conf_timestamp": 2320, "host-ts": 2320}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:83dc71d9-013e-4c66-a2d1-d12cc9adddd0:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:83dc71d9-013e-4c66-a2d1-d12cc9adddd0:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 61eef7c0 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 61eef7c0 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 61eef7c0 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2321 (Fri Jun 22 18:48:20 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2321 (Fri Jun 22 18:48:20 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "703e13db", "local_conf_timestamp": 2321, "host-ts": 2321}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2320 (Fri Jun 22 18:48:20 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2320 (Fri Jun 22 18:48:20 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d771ed7d", "local_conf_timestamp": 2320, "host-ts": 2320}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:0c3f4987-1d01-46dc-aa66-c3d0c6933430:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:0c3f4987-1d01-46dc-aa66-c3d0c6933430:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 64b46030 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 64b46030 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 64b46030 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2331 (Fri Jun 22 18:48:31 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2331 (Fri Jun 22 18:48:31 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "62420b1e", "local_conf_timestamp": 2331, "host-ts": 2331}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2330 (Fri Jun 22 18:48:30 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2331 (Fri Jun 22 18:48:31 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a8757296", "local_conf_timestamp": 2331, "host-ts": 2330}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:baa23295-8069-4395-8756-bc5db853ff9d:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:baa23295-8069-4395-8756-bc5db853ff9d:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 6720458c on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 6720458c on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 6720458c on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2331 (Fri Jun 22 18:48:31 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2331 (Fri Jun 22 18:48:31 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "62420b1e", "local_conf_timestamp": 2331, "host-ts": 2331}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2330 (Fri Jun 22 18:48:30 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2331 (Fri Jun 22 18:48:31 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a8757296", "local_conf_timestamp": 2331, "host-ts": 2330}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:479ef1da-24e7-490e-88ed-c61739a044d0:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:479ef1da-24e7-490e-88ed-c61739a044d0:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 699a90ec on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 699a90ec on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 699a90ec on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2341 (Fri Jun 22 18:48:40 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2341 (Fri Jun 22 18:48:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1d364145", "local_conf_timestamp": 2341, "host-ts": 2341}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2340 (Fri Jun 22 18:48:40 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2340 (Fri Jun 22 18:48:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e7780e73", "local_conf_timestamp": 2340, "host-ts": 2340}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:24152678-5688-4c02-8b24-5f47ba7f2d3c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:24152678-5688-4c02-8b24-5f47ba7f2d3c:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 6c2692ac on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 6c2692ac on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 6c2692ac on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2341 (Fri Jun 22 18:48:40 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2341 (Fri Jun 22 18:48:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1d364145", "local_conf_timestamp": 2341, "host-ts": 2341}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2340 (Fri Jun 22 18:48:40 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2340 (Fri Jun 22 18:48:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e7780e73", "local_conf_timestamp": 2340, "host-ts": 2340}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:36dd77ba-9529-45b4-b449-b7722c0b98fe:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:36dd77ba-9529-45b4-b449-b7722c0b98fe:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 6ebb7244 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 6ebb7244 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 6ebb7244 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2341 (Fri Jun 22 18:48:40 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2341 (Fri Jun 22 18:48:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1d364145", "local_conf_timestamp": 2341, "host-ts": 2341}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2350 (Fri Jun 22 18:48:50 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2350 (Fri Jun 22 18:48:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ef7ba10e", "local_conf_timestamp": 2350, "host-ts": 2350}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:257e02ab-bf9b-4ac2-8ff7-9515d4f1aa56:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:257e02ab-bf9b-4ac2-8ff7-9515d4f1aa56:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 7159a5e8 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 7159a5e8 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 7159a5e8 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2351 (Fri Jun 22 18:48:51 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2352 (Fri Jun 22 18:48:51 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "9643083a", "local_conf_timestamp": 2352, "host-ts": 2351}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2350 (Fri Jun 22 18:48:50 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2350 (Fri Jun 22 18:48:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ef7ba10e", "local_conf_timestamp": 2350, "host-ts": 2350}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:27ec01d4-6761-4468-8ed4-6ab4fe4fcb00:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:27ec01d4-6761-4468-8ed4-6ab4fe4fcb00:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 73bd77e2 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 73bd77e2 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 73bd77e2 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2351 (Fri Jun 22 18:48:51 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2352 (Fri Jun 22 18:48:51 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "9643083a", "local_conf_timestamp": 2352, "host-ts": 2351}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2350 (Fri Jun 22 18:48:50 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2350 (Fri Jun 22 18:48:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ef7ba10e", "local_conf_timestamp": 2350, "host-ts": 2350}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:cc1eb4e6-3fd2-47d2-8998-a0e9e91743f9:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:cc1eb4e6-3fd2-47d2-8998-a0e9e91743f9:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 76ff1456 on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 76ff1456 on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 76ff1456 on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2361 (Fri Jun 22 18:49:00 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2361 (Fri Jun 22 18:49:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "39ce70cf", "local_conf_timestamp": 2361, "host-ts": 2361}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2360 (Fri Jun 22 18:49:00 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2360 (Fri Jun 22 18:49:00 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f77f5089", "local_conf_timestamp": 2360, "host-ts": 2360}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:34564059-b09e-4c89-9a53-9a4a876e15d0:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: end task:34564059-b09e-4c89-9a53-9a4a876e15d0:Get ssh client for lago-he-basic-ansible-suite-master-host-0:\nlago.ssh: DEBUG: Running 796297fe on lago-he-basic-ansible-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 796297fe on lago-he-basic-ansible-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 796297fe on lago-he-basic-ansible-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2361 (Fri Jun 22 18:49:00 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2361 (Fri Jun 22 18:49:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "39ce70cf", "local_conf_timestamp": 2361, "host-ts": 2361}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2360 (Fri Jun 22 18:49:00 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2360 (Fri Jun 22 18:49:00 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f48cce38140>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 139, in <lambda>\n for k, v in _get_he_status(host).items()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-master/ovirt-system-tests/he-basic-ansible-suite-master/test-scenarios/008_restart_he_vm.py", line 83, in _get_he_status\n raise RuntimeError(\'could not parse JSON: %s\' % ret.out)\nRuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2361 (Fri Jun 22 18:49:00 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2361 (Fri Jun 22 18:49:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-0", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "39ce70cf", "local_conf_timestamp": 2361, "host-ts": 2361}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2360 (Fri Jun 22 18:49:00 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2360 (Fri Jun 22 18:49:00 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-ansible-suite-master-host-1", "host-\n--------------------- >> end captured logging << ---------------------'

1 1

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build # 503 - Failure!
by jenkins＠jenkins.phx.ovirt.org 23 Jun '18

23 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/503/ Build Number: 503 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #503 [Sandro Bonazzola] yum-repos: master: adjusted for new packages ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.snapshot_merge Error Message: status: 409 reason: Conflict detail: Cannot create Snapshot. Snapshot is currently being created for VM vm0. Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 68, in wrapper return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/004_basic_sanity.py", line 194, in snapshot_merge api.vms.get(VM0_NAME).snapshots.add(dead_snap2_params) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 34511, in add headers={"Correlation-Id":correlation_id, "Expect":expect} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add return self.request('POST', url, body, headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) RequestError: status: 409 reason: Conflict detail: Cannot create Snapshot. Snapshot is currently being created for VM vm0.

1 2

[oVirt Jenkins] ovirt-system-tests_he-basic-iscsi-suite-master - Build # 266 - Failure!
by jenkins＠jenkins.phx.ovirt.org 22 Jun '18

22 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master… Build Number: 266 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #266 [Sandro Bonazzola] yum-repos: master: adjusted for new packages ----------------- Failed Tests: ----------------- All tests passed

1 2

[JIRA] (OVIRT-2150) missing packages on check-patch
by Dafna Ron (oVirt JIRA) 22 Jun '18

22 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2150?page=com.atlassian.jira.… ] Dafna Ron updated OVIRT-2150: ----------------------------- Issue Type: Bug (was: Task) > missing packages on check-patch > ------------------------------- > > Key: OVIRT-2150 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2150 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_infra, ost_infra_packages > > http://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/1170/c… > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Error: Package: python2-ovsdbapp-0.6.0-1.el7.noarch (alocalsync) > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Requires: python-netaddr > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Error: Package: ovirt-engine-4.3.0-0.0.master.20180604134534.git3e394fa.el7.noarch (alocalsync) > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Requires: pyOpenSSL > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Error: Package: ovirt-provider-ovn-1.2.11-1.el7.noarch (alocalsync) > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Requires: python-netaddr > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-2150) missing packages on check-patch
by Dafna Ron (oVirt JIRA) 22 Jun '18

22 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2150?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2150: ---------------------------------- issue was resolved. > missing packages on check-patch > ------------------------------- > > Key: OVIRT-2150 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2150 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_infra, ost_infra_packages > > http://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/1170/c… > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Error: Package: python2-ovsdbapp-0.6.0-1.el7.noarch (alocalsync) > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Requires: python-netaddr > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Error: Package: ovirt-engine-4.3.0-0.0.master.20180604134534.git3e394fa.el7.noarch (alocalsync) > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Requires: pyOpenSSL > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Error: Package: ovirt-provider-ovn-1.2.11-1.el7.noarch (alocalsync) > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Requires: python-netaddr > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-2150) missing packages on check-patch
by Dafna Ron (oVirt JIRA) 22 Jun '18

22 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2150?page=com.atlassian.jira.… ] Dafna Ron updated OVIRT-2150: ----------------------------- Resolution: Fixed Status: Done (was: To Do) > missing packages on check-patch > ------------------------------- > > Key: OVIRT-2150 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2150 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_infra, ost_infra_packages > > http://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/1170/c… > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Error: Package: python2-ovsdbapp-0.6.0-1.el7.noarch (alocalsync) > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Requires: python-netaddr > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Error: Package: ovirt-engine-4.3.0-0.0.master.20180604134534.git3e394fa.el7.noarch (alocalsync) > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Requires: pyOpenSSL > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Error: Package: ovirt-provider-ovn-1.2.11-1.el7.noarch (alocalsync) > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Requires: python-netaddr > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-2150) missing packages on check-patch
by Dafna Ron (oVirt JIRA) 22 Jun '18

22 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2150?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2150: ---------------------------------- this was resolved. > missing packages on check-patch > ------------------------------- > > Key: OVIRT-2150 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2150 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_infra, ost_infra_packages > > http://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/1170/c… > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Error: Package: python2-ovsdbapp-0.6.0-1.el7.noarch (alocalsync) > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Requires: python-netaddr > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Error: Package: ovirt-engine-4.3.0-0.0.master.20180604134534.git3e394fa.el7.noarch (alocalsync) > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Requires: pyOpenSSL > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Error: Package: ovirt-provider-ovn-1.2.11-1.el7.noarch (alocalsync) > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] Requires: python-netaddr > 10:18:14 [check-patch.basic_suite_master.el7.x86_64] -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-2225) 001_upgrade_engine.test_initialize_engine failed due to missing ovirt-host-deploy pacakge
by Dafna Ron (oVirt JIRA) 22 Jun '18

22 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2225?page=com.atlassian.jira.… ] Dafna Ron updated OVIRT-2225: ----------------------------- Resolution: Fixed Status: Done (was: To Do) > 001_upgrade_engine.test_initialize_engine failed due to missing ovirt-host-deploy pacakge > ----------------------------------------------------------------------------------------- > > Key: OVIRT-2225 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2225 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_infra, ost_infra_packages > > The error we are failing on is: > ("FATAL Internal error (main): [u'ovirt-host-deploy-java-1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch requires ovirt-host-deploy = 1.8.0-0.0.master.20180620111438.git925eabd.el7']",) > Looking at the reposync ovirt-host-deploy-* have a new version except for ovirt-host-deploy itself. > 'Repository google-chrome is listed more than once in the configuration > Repository google-talkplugin is listed more than once in the configuration > ovirt-host-deploy-0:1.8.0-0.0.master.20180531090832.git9811a30.el7.noarch > ovirt-host-deploy-common-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch > ovirt-host-deploy-java-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch > ovirt-host-deploy-javadoc-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch > python2-ovirt-host-deploy-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch > [dron@dron ovirt-system-tests]$ ' > I ran the reposync file to see if the location changed but it did not. I also excluded the package from tested to see it its available somewhere else but its not. > asking Sandro to take a look. > Here are the failed change and build: > For further details about the change see: > https://gerrit.ovirt.org/#/c/92381/1 > For failed test results see: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8293/ -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-2225) 001_upgrade_engine.test_initialize_engine failed due to missing ovirt-host-deploy pacakge
by Dafna Ron (oVirt JIRA) 22 Jun '18

22 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2225?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2225: ---------------------------------- I would not call it a regression :) packages issues caused because ovirt-host location was changed. more infra-packages related I am moving to closed as patch fixed the problem > 001_upgrade_engine.test_initialize_engine failed due to missing ovirt-host-deploy pacakge > ----------------------------------------------------------------------------------------- > > Key: OVIRT-2225 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2225 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_infra, ost_infra_packages > > The error we are failing on is: > ("FATAL Internal error (main): [u'ovirt-host-deploy-java-1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch requires ovirt-host-deploy = 1.8.0-0.0.master.20180620111438.git925eabd.el7']",) > Looking at the reposync ovirt-host-deploy-* have a new version except for ovirt-host-deploy itself. > 'Repository google-chrome is listed more than once in the configuration > Repository google-talkplugin is listed more than once in the configuration > ovirt-host-deploy-0:1.8.0-0.0.master.20180531090832.git9811a30.el7.noarch > ovirt-host-deploy-common-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch > ovirt-host-deploy-java-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch > ovirt-host-deploy-javadoc-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch > python2-ovirt-host-deploy-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch > [dron@dron ovirt-system-tests]$ ' > I ran the reposync file to see if the location changed but it did not. I also excluded the package from tested to see it its available somewhere else but its not. > asking Sandro to take a look. > Here are the failed change and build: > For further details about the change see: > https://gerrit.ovirt.org/#/c/92381/1 > For failed test results see: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8293/ -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-4.2 - Build # 282 - Failure!
by jenkins＠jenkins.phx.ovirt.org 22 Jun '18

22 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/282/ Build Number: 282 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #282 [Sandro Bonazzola] yum-repos: master: adjusted for new packages ----------------- Failed Tests: ----------------- No tests ran.

1 2

[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-master - Build # 122 - Failure!
by jenkins＠jenkins.phx.ovirt.org 22 Jun '18

22 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/122/ Build Number: 122 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #122 [Sandro Bonazzola] yum-repos: master: adjusted for new packages ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.install_cockpit_ovirt Error Message: _install_cockpit_ovirt_on_host(): failed to install cockpit-ovirt-dashboard on host <ovirtlago.virt.HostVM object at 0x7ff6536ee310> -------------------- >> begin captured logging << -------------------- lago.ssh: DEBUG: start task:4045db12-7d51-4e75-98d4-58a330a5477e:Get ssh client for lago-he-node-ng-suite-master-host-1: lago.ssh: DEBUG: start task:498b2fbc-0553-45fd-91e7-43a6d4129d66:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: end task:4045db12-7d51-4e75-98d4-58a330a5477e:Get ssh client for lago-he-node-ng-suite-master-host-1: lago.ssh: DEBUG: end task:498b2fbc-0553-45fd-91e7-43a6d4129d66:Get ssh client for lago-he-node-ng-suite-master-host-0: lago.ssh: DEBUG: Running 73903c02 on lago-he-node-ng-suite-master-host-1: yum -y install cockpit-ovirt-dashboard lago.ssh: DEBUG: Running 739d9744 on lago-he-node-ng-suite-master-host-0: yum -y install cockpit-ovirt-dashboard lago.ssh: DEBUG: Command 73903c02 on lago-he-node-ng-suite-master-host-1 returned with 0 lago.ssh: DEBUG: Command 73903c02 on lago-he-node-ng-suite-master-host-1 output: Loaded plugins: enabled_repos_upload, fastestmirror, imgbased-persist, : package_upload, product-id, search-disabled-repos, subscription- : manager, vdsmupgrade This system is not registered with an entitlement server. You can use subscription-manager to register. Determining fastest mirrors * alocalsync: 192.168.200.1:8585 * ovirt-master-epel: mirrors.develooper.com Nothing to do Uploading Enabled Repositories Report Loaded plugins: fastestmirror, product-id, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. lago.ssh: DEBUG: Command 73903c02 on lago-he-node-ng-suite-master-host-1 errors: https://mirror.hmc.edu/epel/7/x86_64/repodata/d8af7af24cda4f056b3ce6ae7f8c4…: [Errno 14] HTTPS Error 404 - Not Found Trying other mirror. To address this issue please refer to the below wiki article https://wiki.centos.org/yum-errors If above article doesn't help to resolve this issue please use https://bugs.centos.org/. Cannot upload enabled repos report, is this client registered? lago.ssh: DEBUG: Command 739d9744 on lago-he-node-ng-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 739d9744 on lago-he-node-ng-suite-master-host-0 output: Loaded plugins: enabled_repos_upload, fastestmirror, imgbased-persist, : package_upload, product-id, search-disabled-repos, subscription- : manager, vdsmupgrade This system is not registered with an entitlement server. You can use subscription-manager to register. Determining fastest mirrors Uploading Enabled Repositories Report Loaded plugins: fastestmirror, product-id, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. lago.ssh: DEBUG: Command 739d9744 on lago-he-node-ng-suite-master-host-0 errors: BDB2053 Freeing read locks for locker 0x97: 15471/140282900039744 BDB2053 Freeing read locks for locker 0x99: 15471/140282900039744 BDB2053 Freeing read locks for locker 0x9a: 15471/140282900039744 BDB2053 Freeing read locks for locker 0x9b: 15471/140282900039744 One of the configured repositories failed (Unknown), and yum doesn't have enough cached data to continue. At this point the only safe thing yum can do is fail. There are a few ways to work "fix" this: 1. Contact the upstream for the repository and get them to fix the problem. 2. Reconfigure the baseurl/etc. for the repository, to point to a working upstream. This is most often useful if you are using a newer distribution release than is supported by the repository (and the packages for the previous distribution release still work). 3. Run the command with the repository temporarily disabled yum --disablerepo=<repoid> ... 4. Disable the repository permanently, so yum won't use it by default. Yum will then just ignore the repository until you permanently enable it again or use --enablerepo for temporary usage: yum-config-manager --disable <repoid> or subscription-manager repos --disable=<repoid> 5. Configure the failing repository to be skipped, if it is unavailable. Note that yum will try to contact the repo. when it runs most commands, so will have to try and fail each time (and thus. yum will be be much slower). If it is a very temporary problem though, this is often a nice compromise: yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true Cannot retrieve metalink for repository: ovirt-master-epel/x86_64. Please verify its path and try again Cannot upload enabled repos report, is this client registered? lago.utils: ERROR: Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/002_bootstrap.py", line 414, in _install_cockpit_ovirt_on_host nt.assert_equals(ret.code, 0, '_install_cockpit_ovirt_on_host(): failed to install cockpit-ovirt-dashboard on host %s' % host) File "/usr/lib64/python2.7/unittest/case.py", line 553, in assertEqual assertion_func(first, second, msg=msg) File "/usr/lib64/python2.7/unittest/case.py", line 546, in _baseAssertEqual raise self.failureException(msg) AssertionError: _install_cockpit_ovirt_on_host(): failed to install cockpit-ovirt-dashboard on host <ovirtlago.virt.HostVM object at 0x7ff6536ee310> --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/002_bootstrap.py", line 421, in install_cockpit_ovirt nt.assert_true(all(vt.join_all()), 'not all threads finished: %s' % vt) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/002_bootstrap.py", line 414, in _install_cockpit_ovirt_on_host nt.assert_equals(ret.code, 0, '_install_cockpit_ovirt_on_host(): failed to install cockpit-ovirt-dashboard on host %s' % host) File "/usr/lib64/python2.7/unittest/case.py", line 553, in assertEqual assertion_func(first, second, msg=msg) File "/usr/lib64/python2.7/unittest/case.py", line 546, in _baseAssertEqual raise self.failureException(msg) '_install_cockpit_ovirt_on_host(): failed to install cockpit-ovirt-dashboard on host <ovirtlago.virt.HostVM object at 0x7ff6536ee310>\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:4045db12-7d51-4e75-98d4-58a330a5477e:Get ssh client for lago-he-node-ng-suite-master-host-1:\nlago.ssh: DEBUG: start task:498b2fbc-0553-45fd-91e7-43a6d4129d66:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: end task:4045db12-7d51-4e75-98d4-58a330a5477e:Get ssh client for lago-he-node-ng-suite-master-host-1:\nlago.ssh: DEBUG: end task:498b2fbc-0553-45fd-91e7-43a6d4129d66:Get ssh client for lago-he-node-ng-suite-master-host-0:\nlago.ssh: DEBUG: Running 73903c02 on lago-he-node-ng-suite-master-host-1: yum -y install cockpit-ovirt-dashboard\nlago.ssh: DEBUG: Running 739d9744 on lago-he-node-ng-suite-master-host-0: yum -y install cockpit-ovirt-dashboard\nlago.ssh: DEBUG: Command 73903c02 on lago-he-node-ng-suite-master-host-1 returned with 0\nlago.ssh: DEBUG: Command 73903c02 on lago-he-node-ng-suite-master-host-1 output:\n Loaded plugins: enabled_repos_upload, fastestmirror, imgbased-persist,\n : package_upload, product-id, search-disabled-repos, subscription-\n : manager, vdsmupgrade\nThis system is not registered with an entitlement server. You can use subscription-manager to register.\nDetermining fastest mirrors\n * alocalsync: 192.168.200.1:8585\n * ovirt-master-epel: mirrors.develooper.com\nNothing to do\nUploading Enabled Repositories Report\nLoaded plugins: fastestmirror, product-id, subscription-manager\nThis system is not registered with an entitlement server. You can use subscription-manager to register.\n\nlago.ssh: DEBUG: Command 73903c02 on lago-he-node-ng-suite-master-host-1 errors:\n https://mirror.hmc.edu/epel/7/x86_64/repodata/d8af7af24cda4f056b3ce6ae7f8c4…: [Errno 14] HTTPS Error 404 - Not Found\nTrying other mirror.\nTo address this issue please refer to the below wiki article \n\nhttps://wiki.centos.org/yum-errors\n\nIf above article doesn\'t help to resolve this issue please use https://bugs.centos.org/.\n\nCannot upload enabled repos report, is this client registered?\n\nlago.ssh: DEBUG: Command 739d9744 on lago-he-node-ng-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 739d9744 on lago-he-node-ng-suite-master-host-0 output:\n Loaded plugins: enabled_repos_upload, fastestmirror, imgbased-persist,\n : package_upload, product-id, search-disabled-repos, subscription-\n : manager, vdsmupgrade\nThis system is not registered with an entitlement server. You can use subscription-manager to register.\nDetermining fastest mirrors\nUploading Enabled Repositories Report\nLoaded plugins: fastestmirror, product-id, subscription-manager\nThis system is not registered with an entitlement server. You can use subscription-manager to register.\n\nlago.ssh: DEBUG: Command 739d9744 on lago-he-node-ng-suite-master-host-0 errors:\n BDB2053 Freeing read locks for locker 0x97: 15471/140282900039744\nBDB2053 Freeing read locks for locker 0x99: 15471/140282900039744\nBDB2053 Freeing read locks for locker 0x9a: 15471/140282900039744\nBDB2053 Freeing read locks for locker 0x9b: 15471/140282900039744\n\n\n One of the configured repositories failed (Unknown),\n and yum doesn\'t have enough cached data to continue. At this point the only\n safe thing yum can do is fail. There are a few ways to work "fix" this:\n\n 1. Contact the upstream for the repository and get them to fix the problem.\n\n 2. Reconfigure the baseurl/etc. for the repository, to point to a working\n upstream. This is most often useful if you are using a newer\n distribution release than is supported by the repository (and the\n packages for the previous distribution release still work).\n\n 3. Run the command with the repository temporarily disabled\n yum --disablerepo=<repoid> ...\n\n 4. Disable the repository permanently, so yum won\'t use it by default. Yum\n will then just ignore the repository until you permanently enable it\n again or use --enablerepo for temporary usage:\n\n yum-config-manager --disable <repoid>\n or\n subscription-manager repos --disable=<repoid>\n\n 5. Configure the failing repository to be skipped, if it is unavailable.\n Note that yum will try to contact the repo. when it runs most commands,\n so will have to try and fail each time (and thus. yum will be be much\n slower). If it is a very temporary problem though, this is often a nice\n compromise:\n\n yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true\n\nCannot retrieve metalink for repository: ovirt-master-epel/x86_64. Please verify its path and try again\nCannot upload enabled repos report, is this client registered?\n\nlago.utils: ERROR: Error while running thread\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue\n queue.put({\'return\': func()})\n File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-master/ovirt-system-tests/he-node-ng-suite-master/test-scenarios/002_bootstrap.py", line 414, in _install_cockpit_ovirt_on_host\n nt.assert_equals(ret.code, 0, \'_install_cockpit_ovirt_on_host(): failed to install cockpit-ovirt-dashboard on host %s\' % host)\n File "/usr/lib64/python2.7/unittest/case.py", line 553, in assertEqual\n assertion_func(first, second, msg=msg)\n File "/usr/lib64/python2.7/unittest/case.py", line 546, in _baseAssertEqual\n raise self.failureException(msg)\nAssertionError: _install_cockpit_ovirt_on_host(): failed to install cockpit-ovirt-dashboard on host <ovirtlago.virt.HostVM object at 0x7ff6536ee310>\n--------------------- >> end captured logging << ---------------------'

1 1

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #1682
by jenkins＠jenkins.phx.ovirt.org 22 Jun '18

22 Jun '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision ff4050330abc014f5d54d61198050d4b9dc5db3a (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f ff4050330abc014f5d54d61198050d4b9dc5db3a Commit message: "Move repoman project to STDCIv2" > git rev-list --no-walk ff4050330abc014f5d54d61198050d4b9dc5db3a # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins2535441991775911350.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1474, in _commonLoadRepoXML if self._latestRepoXML(local): File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1443, in _latestRepoXML oxml = self._saveOldRepoXML(local) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1300, in _saveOldRepoXML shutil.copy2(local, old_local) File "/usr/lib64/python2.7/shutil.py", line 131, in copy2 copystat(src, dst) File "/usr/lib64/python2.7/shutil.py", line 98, in copystat os.utime(dst, (st.st_atime, st.st_mtime)) OSError: [Errno 2] No such file or directory: '/home/jenkins/mirrors_cache/fedora-updates-fc26/repomd.xml.old.tmp' Build step 'Execute shell' marked build as failure

1 1

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build # 500 - Failure!
by jenkins＠jenkins.phx.ovirt.org 22 Jun '18

22 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/500/ Build Number: 500 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #500 [Sandro Bonazzola] yum-repos: master: adjusted for new packages ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.snapshot_merge Error Message: status: 409 reason: Conflict detail: Cannot remove Snapshot. Snapshot is currently being created for VM vm0. Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 68, in wrapper return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/004_basic_sanity.py", line 200, in snapshot_merge api.vms.get(VM0_NAME).snapshots.list()[-2].delete() File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 34001, in delete headers={"Correlation-Id":correlation_id,"Content-type":None} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 57, in delete return self.request('DELETE', url, body, headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) RequestError: status: 409 reason: Conflict detail: Cannot remove Snapshot. Snapshot is currently being created for VM vm0.

1 2

[oVirt Jenkins] ovirt-system-tests_he-basic-iscsi-suite-master - Build # 264 - Failure!
by jenkins＠jenkins.phx.ovirt.org 21 Jun '18

21 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master… Build Number: 264 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #264 [Sandro Bonazzola] yum-repos: master: adjusted for new packages [Emil Natan] Move repoman project to STDCIv2 ----------------- Failed Tests: ----------------- All tests passed

1 1

[oVirt Jenkins] ovirt-system-tests_he-basic-suite-4.2 - Build # 310 - Failure!
by jenkins＠jenkins.phx.ovirt.org 21 Jun '18

21 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.2/310/ Build Number: 310 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #310 [Sandro Bonazzola] yum-repos: master: adjusted for new packages [Emil Natan] Move repoman project to STDCIv2 ----------------- Failed Tests: ----------------- All tests passed

1 1

[JIRA] (OVIRT-2221) Reposync builder will no work with DNF
by Gal Ben Haim (oVirt JIRA) 21 Jun '18

21 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2221?page=com.atlassian.jira.… ] Gal Ben Haim updated OVIRT-2221: -------------------------------- Epic Link: OVIRT-2177 > Reposync builder will no work with DNF > --------------------------------------- > > Key: OVIRT-2221 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2221 > Project: oVirt - virtualization made easy > Issue Type: Task > Components: OST > Reporter: Gal Ben Haim > Assignee: infra > > Part of the reposync builder tool is a Yum plugin which auto-generates the reposync-config with "includepkgs" field. > DNF doesn't support Yum plugins, so we will have to rewrite the plugin for DNF, -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-2221) Reposync builder will no work with DNF
by Gal Ben Haim (oVirt JIRA) 21 Jun '18

21 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2221?page=com.atlassian.jira.… ] Gal Ben Haim updated OVIRT-2221: -------------------------------- Epic Link: OVIRT-2177 > Reposync builder will no work with DNF > --------------------------------------- > > Key: OVIRT-2221 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2221 > Project: oVirt - virtualization made easy > Issue Type: Task > Components: OST > Reporter: Gal Ben Haim > Assignee: infra > > Part of the reposync builder tool is a Yum plugin which auto-generates the reposync-config with "includepkgs" field. > DNF doesn't support Yum plugins, so we will have to rewrite the plugin for DNF, -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-2221) Reposync builder will no work with DNF
by Gal Ben Haim (oVirt JIRA) 21 Jun '18

21 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2221?page=com.atlassian.jira.… ] Gal Ben Haim updated OVIRT-2221: -------------------------------- Status: In Progress (was: To Do) > Reposync builder will no work with DNF > --------------------------------------- > > Key: OVIRT-2221 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2221 > Project: oVirt - virtualization made easy > Issue Type: Task > Components: OST > Reporter: Gal Ben Haim > Assignee: infra > > Part of the reposync builder tool is a Yum plugin which auto-generates the reposync-config with "includepkgs" field. > DNF doesn't support Yum plugins, so we will have to rewrite the plugin for DNF, -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-2229) Automate updating of ovirt-engine-wildfly
by eyal edri (oVirt JIRA) 21 Jun '18

21 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2229?page=com.atlassian.jira.… ] eyal edri commented on OVIRT-2229: ---------------------------------- cc [~mwperina] we're looking into a way to automate the build of wildfly from their nightly builds. > Automate updating of ovirt-engine-wildfly > ----------------------------------------- > > Key: OVIRT-2229 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2229 > Project: oVirt - virtualization made easy > Issue Type: New Feature > Components: CI client projects > Reporter: Barak Korren > Assignee: infra > > As an oVirt developer I want to test the oVirt source code against new versions of Wildfly as early as possible so that I can find issues early. > h3. Acceptance Criteria: > # Have a 'poll' job monitor for releases of Wildfly and update the ovirt-engine-wildfly source code. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-2229) Automate updating of ovirt-engine-wildfly
by Barak Korren (oVirt JIRA) 21 Jun '18

21 Jun '18

Barak Korren created OVIRT-2229: ----------------------------------- Summary: Automate updating of ovirt-engine-wildfly Key: OVIRT-2229 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2229 Project: oVirt - virtualization made easy Issue Type: New Feature Components: CI client projects Reporter: Barak Korren Assignee: infra As an oVirt developer I want to test the oVirt source code against new versions of Wildfly as early as possible so that I can find issues early. h3. Acceptance Criteria: # Have a 'poll' job monitor for releases of Wildfly and update the ovirt-engine-wildfly source code. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-2228) automating repoman deployment
by Emil Natan (oVirt JIRA) 21 Jun '18

21 Jun '18

Emil Natan created OVIRT-2228: --------------------------------- Summary: automating repoman deployment Key: OVIRT-2228 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2228 Project: oVirt - virtualization made easy Issue Type: Task Reporter: Emil Natan Assignee: infra Automate repoman deployment using its own change-queue or a 'ci-tools' change-queue. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-4.2 - Build # 280 - Failure!
by jenkins＠jenkins.phx.ovirt.org 21 Jun '18

21 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/280/ Build Number: 280 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #280 [Sandro Bonazzola] yum-repos: master: adjusted for new packages ----------------- Failed Tests: ----------------- No tests ran.

1 1

[JIRA] (OVIRT-2227) Re: Push rights to vdsm
by eyal edri (oVirt JIRA) 21 Jun '18

21 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2227?page=com.atlassian.jira.… ] eyal edri commented on OVIRT-2227: ---------------------------------- [~ena(a)redhat.com] this request was to be able to send a patch to VDSM on Gerrit, not to trigger CI jobs on a patch. Sending patch to Gerrit can be done by any registered user of Gerrit ( anyone can register, even malicious users for that matter ), and that is why we have the whitelist in effect, so only approved users will be able to trigger jobs on the CI. Having said that, he will need to be added to the whitelist if he wants to run CI jobs, so maybe rename the ticket accordingly since the original issue was resolved AFAIK > Re: Push rights to vdsm > ----------------------- > > Key: OVIRT-2227 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2227 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Emil Natan > Assignee: infra > > Hi > You just need to ask one of the existing maintainers to send a patch to the > 'jenkins-whitelist' repo on gerrit.ovirt.org and > one of the infra members can merge it once it has +1 from an existing > maintainer. > I'm also adding 'infra-support' to open a ticket. > On Thu, Jun 21, 2018 at 11:14 AM, Tomasz Barański <tbaransk(a)redhat.com> > wrote: > > Eyal, > > > > Apparently I also need to be added to the CI whitelist. > > > > Tomo > > > > On 20.06.2018 12:49, Eyal Edri wrote: > > > > No worries, glad it works for you. > > Don't hesitate to reach out to infra again if you have any issues. > > > > On Wed, Jun 20, 2018 at 1:29 PM Tomasz Barański <tbaransk(a)redhat.com> > > wrote: > > > >> Eyal, > >> > >> I didn't set `username` in gerrit. Seems to have missed this step. > >> > >> It works now, sorry to bother you. > >> > >> Tomo > >> > >> On 20.06.2018 12:03, Eyal Edri wrote: > >> > >> Any registered user have push rights to new patches, did you follow the > >> guidelines on how to contribute to and oVirt project? > >> > >> https://ovirt.org/develop/dev-process/working-with-gerrit/ > >> > >> On Wed, Jun 20, 2018 at 1:00 PM Tomasz Barański <tbaransk(a)redhat.com> > >> wrote: > >> > >>> Hello, > >>> > >>> As a new Red Hatter, I need push right to `vdsm` project in oVirt's > >>> gerrit. > >>> > >>> Thank you > >>> Tomasz > >>> _______________________________________________ > >>> Infra mailing list -- infra(a)ovirt.org > >>> To unsubscribe send an email to infra-leave(a)ovirt.org > >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > >>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community- > >>> guidelines/ > >>> List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/ > >>> message/2QDBTW4HH2FUGSQXHVDML5K3SIEAKIE2/ > >>> > >> > >> > >> -- > >> > >> Eyal edri > >> > >> > >> MANAGER > >> > >> RHV DevOps > >> > >> EMEA VIRTUALIZATION R&D > >> > >> > >> Red Hat EMEA <https://www.redhat.com/> > >> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > >> phone: +972-9-7692018 > >> irc: eedri (on #tlv #rhev-dev #rhev-integ) > >> > >> > >> > > > > -- > > > > Eyal edri > > > > > > MANAGER > > > > RHV DevOps > > > > EMEA VIRTUALIZATION R&D > > > > > > Red Hat EMEA <https://www.redhat.com/> > > <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > > phone: +972-9-7692018 > > irc: eedri (on #tlv #rhev-dev #rhev-integ) > > > > > > > > _______________________________________________ > > Infra mailing list -- infra(a)ovirt.org > > To unsubscribe send an email to infra-leave(a)ovirt.org > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > oVirt Code of Conduct: https://www.ovirt.org/community/about/community- > > guidelines/ > > List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/ > > message/F5LTNHLWRG4MLLBKMJQQGAOFDUGSPPCO/ > > > > > -- > Emil Natan > RHV DevOps -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-2227) Re: Push rights to vdsm
by Emil Natan (oVirt JIRA) 21 Jun '18

21 Jun '18

Emil Natan created OVIRT-2227: --------------------------------- Summary: Re: Push rights to vdsm Key: OVIRT-2227 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2227 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Emil Natan Assignee: infra Hi You just need to ask one of the existing maintainers to send a patch to the 'jenkins-whitelist' repo on gerrit.ovirt.org and one of the infra members can merge it once it has +1 from an existing maintainer. I'm also adding 'infra-support' to open a ticket. On Thu, Jun 21, 2018 at 11:14 AM, Tomasz Barański <tbaransk(a)redhat.com> wrote: > Eyal, > > Apparently I also need to be added to the CI whitelist. > > Tomo > > On 20.06.2018 12:49, Eyal Edri wrote: > > No worries, glad it works for you. > Don't hesitate to reach out to infra again if you have any issues. > > On Wed, Jun 20, 2018 at 1:29 PM Tomasz Barański <tbaransk(a)redhat.com> > wrote: > >> Eyal, >> >> I didn't set `username` in gerrit. Seems to have missed this step. >> >> It works now, sorry to bother you. >> >> Tomo >> >> On 20.06.2018 12:03, Eyal Edri wrote: >> >> Any registered user have push rights to new patches, did you follow the >> guidelines on how to contribute to and oVirt project? >> >> https://ovirt.org/develop/dev-process/working-with-gerrit/ >> >> On Wed, Jun 20, 2018 at 1:00 PM Tomasz Barański <tbaransk(a)redhat.com> >> wrote: >> >>> Hello, >>> >>> As a new Red Hatter, I need push right to `vdsm` project in oVirt's >>> gerrit. >>> >>> Thank you >>> Tomasz >>> _______________________________________________ >>> Infra mailing list -- infra(a)ovirt.org >>> To unsubscribe send an email to infra-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community- >>> guidelines/ >>> List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/ >>> message/2QDBTW4HH2FUGSQXHVDML5K3SIEAKIE2/ >>> >> >> >> -- >> >> Eyal edri >> >> >> MANAGER >> >> RHV DevOps >> >> EMEA VIRTUALIZATION R&D >> >> >> Red Hat EMEA <https://www.redhat.com/> >> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >> phone: +972-9-7692018 >> irc: eedri (on #tlv #rhev-dev #rhev-integ) >> >> >> > > -- > > Eyal edri > > > MANAGER > > RHV DevOps > > EMEA VIRTUALIZATION R&D > > > Red Hat EMEA <https://www.redhat.com/> > <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) > > > > _______________________________________________ > Infra mailing list -- infra(a)ovirt.org > To unsubscribe send an email to infra-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community- > guidelines/ > List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/ > message/F5LTNHLWRG4MLLBKMJQQGAOFDUGSPPCO/ > > -- Emil Natan RHV DevOps -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

Push rights to vdsm
by Tomasz Barański 21 Jun '18

21 Jun '18

Hello, As a new Red Hatter, I need push right to `vdsm` project in oVirt's gerrit. Thank you Tomasz

3 5

[JIRA] (OVIRT-2226) Use nested stages to improve STDCI/CQ BlueOcean output
by Barak Korren (oVirt JIRA) 21 Jun '18

21 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2226?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-2226: -------------------------------- Description: "Nested stages is something we wanted for a while, since without them its impossible to logically split apart operations that happen on the same node within a pipeline parallel branch. We were not the only ones to wat this, so this was tracked upstream in [JENKINS-38442|https://issues.jenkins-ci.org/browse/JENKINS-38442]\r\n\r\nA [PR|https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/227] implementing some support for this was finally merged into the [pipeline-model-definition-plugin|https://github.com/jenkinsci/pipeline-mode…] and included in the [1.3|https://github.com/jenkinsci/pipeline-model-definition-plugin/commits/p…] release.\r\n\r\nThe PR seems to indicate this may only be supported in the declarative pipeline syntax, so we'll need to check if this can also work with the iterative syntax or otherwise switch our code to the declarative syntax if possible." (was: Nested stages is something we wanted for a while, since without them its impossible to logically split apart operations that happen on the same node within a pipeline parallel branch. We were not the only ones to wat this, so this was tracked upstream in [JENKINS-38442|https://issues.jenkins-ci.org/browse/JENKINS-38442] A [PR|https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/227] implementing some support for this was finally merged into the [pipeline-model-definition-plugin|https://github.com/jenkinsci/pipeline-mode…] and included in the [1.3|https://github.com/jenkinsci/pipeline-model-definition-plugin/commits/p…] The PR seems to indicate this may only be supported in the declarative pipeline syntax, so we'll need to check if this can also work with the iterative syntax or otherwise switch our code to the declarative syntax if possible.) > Use nested stages to improve STDCI/CQ BlueOcean output > ------------------------------------------------------ > > Key: OVIRT-2226 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2226 > Project: oVirt - virtualization made easy > Issue Type: Epic > Components: Change Queue, Standard CI (Pipelines) > Reporter: Barak Korren > Assignee: infra > > Nested stages is something we wanted for a while, since without them its impossible to logically split apart operations that happen on the same node within a pipeline parallel branch. We were not the only ones to wat this, so this was tracked upstream in [JENKINS-38442|https://issues.jenkins-ci.org/browse/JENKINS-38442] > A [PR|https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/227] implementing some support for this was finally merged into the [pipeline-model-definition-plugin|https://github.com/jenkinsci/pipeline-mode…] and included in the [1.3|https://github.com/jenkinsci/pipeline-model-definition-plugin/commits/p…] release. > The PR seems to indicate this may only be supported in the declarative pipeline syntax, so we'll need to check if this can also work with the iterative syntax or otherwise switch our code to the declarative syntax if possible. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-2226) Use nested stages to improve STDCI/CQ BlueOcean output
by Barak Korren (oVirt JIRA) 21 Jun '18

21 Jun '18

Barak Korren created OVIRT-2226: ----------------------------------- Summary: Use nested stages to improve STDCI/CQ BlueOcean output Key: OVIRT-2226 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2226 Project: oVirt - virtualization made easy Issue Type: Epic Components: Change Queue, Standard CI (Pipelines) Reporter: Barak Korren Assignee: infra Nested stages is something we wanted for a while, since without them its impossible to logically split apart operations that happen on the same node within a pipeline parallel branch. We were not the only ones to wat this, so this was tracked upstream in [JENKINS-38442|https://issues.jenkins-ci.org/browse/JENKINS-38442] A [PR|https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/227] implementing some support for this was finally merged into the [pipeline-model-definition-plugin|https://github.com/jenkinsci/pipeline-mode…] and included in the [1.3|https://github.com/jenkinsci/pipeline-model-definition-plugin/commits/p…] The PR seems to indicate this may only be supported in the declarative pipeline syntax, so we'll need to check if this can also work with the iterative syntax or otherwise switch our code to the declarative syntax if possible. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #1679
by jenkins＠jenkins.phx.ovirt.org 21 Jun '18

21 Jun '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 7e8a988cfa3a5cde022712ab4142923ebffbb4f4 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 7e8a988cfa3a5cde022712ab4142923ebffbb4f4 Commit message: "Move get_generic_queue_build_args to stdci_project" > git rev-list --no-walk 7e8a988cfa3a5cde022712ab4142923ebffbb4f4 # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins8105867571467064723.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7' Build step 'Execute shell' marked build as failure

1 1

[JIRA] (OVIRT-2225) 001_upgrade_engine.test_initialize_engine failed due to missing ovirt-host-deploy pacakge
by Barak Korren (oVirt JIRA) 21 Jun '18

21 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2225?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-2225: ------------------------------------- so basically this is an ovirt-engine regression and the specfile needx to be fixed? > 001_upgrade_engine.test_initialize_engine failed due to missing ovirt-host-deploy pacakge > ----------------------------------------------------------------------------------------- > > Key: OVIRT-2225 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2225 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_infra, ost_infra_packages > > The error we are failing on is: > ("FATAL Internal error (main): [u'ovirt-host-deploy-java-1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch requires ovirt-host-deploy = 1.8.0-0.0.master.20180620111438.git925eabd.el7']",) > Looking at the reposync ovirt-host-deploy-* have a new version except for ovirt-host-deploy itself. > 'Repository google-chrome is listed more than once in the configuration > Repository google-talkplugin is listed more than once in the configuration > ovirt-host-deploy-0:1.8.0-0.0.master.20180531090832.git9811a30.el7.noarch > ovirt-host-deploy-common-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch > ovirt-host-deploy-java-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch > ovirt-host-deploy-javadoc-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch > python2-ovirt-host-deploy-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch > [dron@dron ovirt-system-tests]$ ' > I ran the reposync file to see if the location changed but it did not. I also excluded the package from tested to see it its available somewhere else but its not. > asking Sandro to take a look. > Here are the failed change and build: > For further details about the change see: > https://gerrit.ovirt.org/#/c/92381/1 > For failed test results see: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8293/ -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[oVirt Jenkins] ovirt-system-tests_he-basic-iscsi-suite-master - Build # 262 - Failure!
by jenkins＠jenkins.phx.ovirt.org 21 Jun '18

21 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master… Build Number: 262 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #262 [Sandro Bonazzola] yum-repos: master: adjusted for new packages [Daniel Belenky] Move get_generic_queue_build_args to stdci_project ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.add_secondary_storage_domains Error Message: Fault reason is "Operation Failed". Fault detail is "[Cannot add storage server connection when Host status is not up]". HTTP response code is 409. -------------------- >> begin captured logging << -------------------- lago.utils: ERROR: Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 491, in add_nfs_storage_domain add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, SD_NFS_PATH) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 496, in add_generic_nfs_storage_domain add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 552, in add_generic_nfs_storage_domain_4 _add_storage_domain_4(api, p) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 447, in _add_storage_domain_4 sd = sds_service.add(p) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 24011, in add return self._internal_add(storage_domain, headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault self._raise_error(response, body) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error Error: Fault reason is "Operation Failed". Fault detail is "[Cannot add storage server connection when Host status is not up]". HTTP response code is 409. --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 577, in add_secondary_storage_domains vt.join_all() File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 491, in add_nfs_storage_domain add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, SD_NFS_PATH) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 496, in add_generic_nfs_storage_domain add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 552, in add_generic_nfs_storage_domain_4 _add_storage_domain_4(api, p) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 447, in _add_storage_domain_4 sd = sds_service.add(p) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 24011, in add return self._internal_add(storage_domain, headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault self._raise_error(response, body) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error 'Fault reason is "Operation Failed". Fault detail is "[Cannot add storage server connection when Host status is not up]". HTTP response code is 409.\n-------------------- >> begin captured logging << --------------------\nlago.utils: ERROR: Error while running thread\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue\n queue.put({\'return\': func()})\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 491, in add_nfs_storage_domain\n add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, SD_NFS_PATH)\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 496, in add_generic_nfs_storage_domain\n add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 552, in add_generic_nfs_storage_domain_4\n _add_storage_domain_4(api, p)\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 447, in _add_storage_domain_4\n sd = sds_service.add(p)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 24011, in add\n return self._internal_add(storage_domain, headers, query, wait)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add\n return future.wait() if wait else future\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait\n return self._code(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback\n self._check_fault(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault\n self._raise_error(response, body)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error\n raise error\nError: Fault reason is "Operation Failed". Fault detail is "[Cannot add storage server connection when Host status is not up]". HTTP response code is 409.\n--------------------- >> end captured logging << ---------------------'

1 1

[oVirt Jenkins] ovirt-system-tests_network-suite-master - Build # 319 - Failure!
by jenkins＠jenkins.phx.ovirt.org 21 Jun '18

21 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-master/319/ Build Number: 319 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #319 [Milan Zamazal] Test snapshot with memory in 4.2 [Daniel Belenky] Move get_generic_queue_build_args to stdci_project ----------------- Failed Tests: ----------------- No tests ran.

1 1

[oVirt Jenkins] ovirt-system-tests_performance-suite-master - Build # 419 - Failure!
by jenkins＠jenkins.phx.ovirt.org 21 Jun '18

21 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_performance-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_performance-suite-master/41… Build Number: 419 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #419 [Milan Zamazal] Test snapshot with memory in 4.2 [Daniel Belenky] Move get_generic_queue_build_args to stdci_project ----------------- Failed Tests: ----------------- No tests ran.

1 1

[oVirt Jenkins] ovirt-system-tests_he-basic-ansible-suite-master - Build # 372 - Failure!
by jenkins＠jenkins.phx.ovirt.org 21 Jun '18

21 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-mast… Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-mast… Build Number: 372 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #372 [Gal Ben Haim] upgrade_from_release_suite_master Update pre-reposync [Daniel Belenky] Install python-six during global-setup [Barak Korren] Make mock_runner use curl instead of wget [Barak Korren] Fix proxied configuration for FC27 and FC28 ----------------- Failed Tests: ----------------- All tests passed

1 2

oVirt infra weekly meeting notes 20.06.2018
by Evgheni Dereveanchin 20 Jun '18

20 Jun '18

Hi everyone, Please find the topics of this week's infra meeting below: *PHX - FC26 slaves replaced with FC28- GID issue with mock OVIRT-2210- python-six issue OVIRT-2218- dnf-makecache patch ready for merge OVIRT-1927- CVE patching - ticket opened to patch el7 slaves- Puppet check-patch migration to STDCI re-started- Monitoring migration - access to PHX open form new box OVIRT-502- CentOS bare metal sharing - port access requested OVIRT-2179* -- Regards, Evgheni Dereveanchin

1 0

[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-master - Build # 118 - Failure!
by jenkins＠jenkins.phx.ovirt.org 20 Jun '18

20 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/118/ Build Number: 118 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #118 [Milan Zamazal] Test snapshot with memory in 4.2 [Daniel Belenky] Install python-six during global-setup [Barak Korren] Make mock_runner use curl instead of wget [Barak Korren] Fix proxied configuration for FC27 and FC28 [Daniel Belenky] Move project functions to a shared lib [Daniel Belenky] Utilize stdci_project groovy lib in CQ [Daniel Belenky] Move get_generic_queue_build_args to stdci_project ----------------- Failed Tests: ----------------- All tests passed

1 1

[oVirt Jenkins] ovirt-system-tests_he-basic-suite-4.2 - Build # 307 - Failure!
by jenkins＠jenkins.phx.ovirt.org 20 Jun '18

20 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.2/307/ Build Number: 307 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #307 [Gal Ben Haim] upgrade_from_release_suite_master Update pre-reposync [Daniel Belenky] Install python-six during global-setup [Barak Korren] Make mock_runner use curl instead of wget [Barak Korren] Fix proxied configuration for FC27 and FC28 [Daniel Belenky] Move project functions to a shared lib [Daniel Belenky] Utilize stdci_project groovy lib in CQ ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 008_restart_he_vm.restart_he_vm Error Message: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1959 (Wed Jun 20 06:02:22 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=1960 (Wed Jun 20 06:02:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "66b6e371", "local_conf_timestamp": 1960, "host-ts": 1959}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1956 (Wed Jun 20 06:02:18 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1956 (Wed Jun 20 06:02:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "en -------------------- >> begin captured logging << -------------------- lago.ssh: DEBUG: start task:7fa0b63b-1d82-4f26-b3b0-43a232ee42f2:Get ssh client for lago-he-basic-suite-4-2-host-1: lago.ssh: DEBUG: end task:7fa0b63b-1d82-4f26-b3b0-43a232ee42f2:Get ssh client for lago-he-basic-suite-4-2-host-1: lago.ssh: DEBUG: Running a8297fbc on lago-he-basic-suite-4-2-host-1: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a8297fbc on lago-he-basic-suite-4-2-host-1 returned with 0 lago.ssh: DEBUG: Command a8297fbc on lago-he-basic-suite-4-2-host-1 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1797 (Wed Jun 20 05:59:40 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1797 (Wed Jun 20 05:59:40 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8ca7d5c5", "local_conf_timestamp": 1797, "host-ts": 1797}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1796 (Wed Jun 20 05:59:39 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1797 (Wed Jun 20 05:59:40 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6e153d34", "local_conf_timestamp": 1797, "host-ts": 1796}, "global_maintenance": true} root: INFO: * Shutting down HE VM on host: lago-he-basic-suite-4-2-host-0 lago.ssh: DEBUG: start task:5a3e26ae-7a8e-4aa7-936d-05bac937aebb:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:5a3e26ae-7a8e-4aa7-936d-05bac937aebb:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running a8b75fd0 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-shutdown lago.ssh: DEBUG: Command a8b75fd0 on lago-he-basic-suite-4-2-host-0 returned with 0 root: INFO: * Command succeeded root: INFO: * Waiting for VM to be down... lago.ssh: DEBUG: start task:8798cb39-f09b-4e2e-bc19-6e4b12083bd5:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:8798cb39-f09b-4e2e-bc19-6e4b12083bd5:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running a9444742 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a9444742 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command a9444742 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1797 (Wed Jun 20 05:59:40 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1797 (Wed Jun 20 05:59:40 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8ca7d5c5", "local_conf_timestamp": 1797, "host-ts": 1797}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1796 (Wed Jun 20 05:59:39 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1797 (Wed Jun 20 05:59:40 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6e153d34", "local_conf_timestamp": 1797, "host-ts": 1796}, "global_maintenance": false} lago.ssh: DEBUG: start task:18301a7b-63cf-479c-a7cd-abc56f300691:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:18301a7b-63cf-479c-a7cd-abc56f300691:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running b00ee8e8 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b00ee8e8 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command b00ee8e8 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1807 (Wed Jun 20 05:59:51 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1809 (Wed Jun 20 05:59:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e73d4537", "local_conf_timestamp": 1809, "host-ts": 1807}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1806 (Wed Jun 20 05:59:49 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1806 (Wed Jun 20 05:59:49 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "516e6745", "local_conf_timestamp": 1806, "host-ts": 1806}, "global_maintenance": true} lago.ssh: DEBUG: start task:74ea2f58-62fe-48d7-879f-ccaa247d0b69:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:74ea2f58-62fe-48d7-879f-ccaa247d0b69:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running b276bc8c on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b276bc8c on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command b276bc8c on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1817 (Wed Jun 20 06:00:00 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=1817 (Wed Jun 20 06:00:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e573deaf", "local_conf_timestamp": 1817, "host-ts": 1817}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1816 (Wed Jun 20 05:59:58 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1816 (Wed Jun 20 05:59:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "43127f80", "local_conf_timestamp": 1816, "host-ts": 1816}, "global_maintenance": true} lago.ssh: DEBUG: start task:eafbaa04-76f1-4346-bbf2-f5154b162e05:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:eafbaa04-76f1-4346-bbf2-f5154b162e05:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running b4c932bc on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b4c932bc on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command b4c932bc on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1817 (Wed Jun 20 06:00:00 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=1817 (Wed Jun 20 06:00:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e573deaf", "local_conf_timestamp": 1817, "host-ts": 1817}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1816 (Wed Jun 20 05:59:58 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1816 (Wed Jun 20 05:59:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "43127f80", "local_conf_timestamp": 1816, "host-ts": 1816}, "global_maintenance": true} lago.ssh: DEBUG: start task:7e37932e-79d8-4fb1-b018-e509c1be2079:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:7e37932e-79d8-4fb1-b018-e509c1be2079:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running b71e9f16 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b71e9f16 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command b71e9f16 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1817 (Wed Jun 20 06:00:00 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=1817 (Wed Jun 20 06:00:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e573deaf", "local_conf_timestamp": 1817, "host-ts": 1817}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1816 (Wed Jun 20 05:59:58 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1816 (Wed Jun 20 05:59:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "43127f80", "local_conf_timestamp": 1816, "host-ts": 1816}, "global_maintenance": true} lago.ssh: DEBUG: start task:6721a1b8-c0be-4589-aa6f-06d4593145d7:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:6721a1b8-c0be-4589-aa6f-06d4593145d7:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running b97a0c64 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b97a0c64 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command b97a0c64 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1827 (Wed Jun 20 06:00:10 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1827 (Wed Jun 20 06:00:10 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "82f1748b", "local_conf_timestamp": 1827, "host-ts": 1827}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1826 (Wed Jun 20 06:00:09 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1827 (Wed Jun 20 06:00:09 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "02916659", "local_conf_timestamp": 1827, "host-ts": 1826}, "global_maintenance": true} root: INFO: * VM is down. root: INFO: * Restarting services... lago.ssh: DEBUG: start task:f277519c-ff1c-4145-b766-1529193f9736:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:f277519c-ff1c-4145-b766-1529193f9736:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running ba107a50 on lago-he-basic-suite-4-2-host-0: systemctl restart vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command ba107a50 on lago-he-basic-suite-4-2-host-0 returned with 0 root: INFO: * Success. root: INFO: * Waiting for agent to be ready... lago.ssh: DEBUG: start task:4720df71-76bb-44f1-b960-8d29c77e7eeb:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:4720df71-76bb-44f1-b960-8d29c77e7eeb:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running c5a97c9a on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command c5a97c9a on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command c5a97c9a on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:31ff44a2-deb7-4f53-af62-ada0d3de7814:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:31ff44a2-deb7-4f53-af62-ada0d3de7814:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running cc151d64 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command cc151d64 on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command cc151d64 on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:dafa50c4-5878-45d5-baf2-f29137211116:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:dafa50c4-5878-45d5-baf2-f29137211116:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running ce3e3648 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command ce3e3648 on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command ce3e3648 on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:c67b6f95-f884-4671-ab08-58e01a0e0ada:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:c67b6f95-f884-4671-ab08-58e01a0e0ada:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running d082176c on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command d082176c on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command d082176c on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:1a9235d2-a03f-445b-b0c0-bc2e74f552a2:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:1a9235d2-a03f-445b-b0c0-bc2e74f552a2:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running d2aad948 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command d2aad948 on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command d2aad948 on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:6b6ea68c-dd38-4096-a0ac-dd38a55e2688:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:6b6ea68c-dd38-4096-a0ac-dd38a55e2688:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running d4d59258 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command d4d59258 on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command d4d59258 on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:b5a78ba3-a0fa-4cdf-ac2d-f1ebd5340516:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:b5a78ba3-a0fa-4cdf-ac2d-f1ebd5340516:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running d710521a on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command d710521a on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command d710521a on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:d74eb21b-e2d0-48af-95a6-6ae0d9c322b2:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:d74eb21b-e2d0-48af-95a6-6ae0d9c322b2:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running d93d74f0 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command d93d74f0 on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command d93d74f0 on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:d5883737-42de-4bd9-b14c-3aa39c8f2dfd:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:d5883737-42de-4bd9-b14c-3aa39c8f2dfd:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running db7573b2 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command db7573b2 on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command db7573b2 on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:695e26b3-2326-47f0-81e6-8ba946e0eab9:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:695e26b3-2326-47f0-81e6-8ba946e0eab9:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running dd92f5b6 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command dd92f5b6 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command dd92f5b6 on lago-he-basic-suite-4-2-host-0 output: !! Cluster is in GLOBAL MAINTENANCE mode !! --== Host 1 status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : lago-he-basic-suite-4-2-host-0.lago.local Host ID : 1 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"} Score : 0 stopped : False Local maintenance : False crc32 : 612645ed local_conf_timestamp : 1889 Host timestamp : 1889 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=1889 (Wed Jun 20 06:01:12 2018) host-id=1 score=0 vm_conf_refresh_time=1889 (Wed Jun 20 06:01:13 2018) conf_on_shared_storage=True maintenance=False state=ReinitializeFSM stopped=False --== Host 2 status ==-- conf_on_shared_storage : True Status up-to-date : False Hostname : lago-he-basic-suite-4-2-host-1 Host ID : 2 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : c28ea16d local_conf_timestamp : 1886 Host timestamp : 1886 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=1886 (Wed Jun 20 06:01:09 2018) host-id=2 score=3400 vm_conf_refresh_time=1886 (Wed Jun 20 06:01:09 2018) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False !! Cluster is in GLOBAL MAINTENANCE mode !! root: INFO: * Agent is ready. root: INFO: * Starting VM... lago.ssh: DEBUG: start task:34e67d6d-becb-4934-b953-d59e64f2a55c:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:34e67d6d-becb-4934-b953-d59e64f2a55c:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running de2286cc on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-start lago.ssh: DEBUG: Command de2286cc on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command de2286cc on lago-he-basic-suite-4-2-host-0 output: VM exists and is down, cleaning up and restarting root: INFO: * Command succeeded root: INFO: * Waiting to VM to be UP... lago.ssh: DEBUG: start task:aa25364d-1a02-4256-82d9-50f4799364b3:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:aa25364d-1a02-4256-82d9-50f4799364b3:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running df8a2718 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command df8a2718 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command df8a2718 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1889 (Wed Jun 20 06:01:12 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=1889 (Wed Jun 20 06:01:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=ReinitializeFSM\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "612645ed", "local_conf_timestamp": 1889, "host-ts": 1889}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1886 (Wed Jun 20 06:01:09 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1886 (Wed Jun 20 06:01:09 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c28ea16d", "local_conf_timestamp": 1886, "host-ts": 1886}, "global_maintenance": true} lago.ssh: DEBUG: start task:807fa695-4172-4455-961d-5aab46991388:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:807fa695-4172-4455-961d-5aab46991388:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running e611a3a4 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e611a3a4 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command e611a3a4 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1899 (Wed Jun 20 06:01:22 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1899 (Wed Jun 20 06:01:22 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e627649f", "local_conf_timestamp": 1899, "host-ts": 1899}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1896 (Wed Jun 20 06:01:18 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1896 (Wed Jun 20 06:01:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d0f2b9a8", "local_conf_timestamp": 1896, "host-ts": 1896}, "global_maintenance": true} root: INFO: * VM is UP. root: INFO: * Waiting for engine to start ... lago.ssh: DEBUG: start task:dd69f784-ffa7-437e-8146-4b4550768f6f:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:dd69f784-ffa7-437e-8146-4b4550768f6f:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running e69adf3e on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e69adf3e on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command e69adf3e on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1899 (Wed Jun 20 06:01:22 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1899 (Wed Jun 20 06:01:22 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e627649f", "local_conf_timestamp": 1899, "host-ts": 1899}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1896 (Wed Jun 20 06:01:18 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1896 (Wed Jun 20 06:01:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d0f2b9a8", "local_conf_timestamp": 1896, "host-ts": 1896}, "global_maintenance": true} lago.ssh: DEBUG: start task:73996669-0bfe-4f86-a81d-ee17c02c5b7c:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:73996669-0bfe-4f86-a81d-ee17c02c5b7c:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running ed261ac6 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ed261ac6 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command ed261ac6 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1909 (Wed Jun 20 06:01:32 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1910 (Wed Jun 20 06:01:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "06938df0", "local_conf_timestamp": 1910, "host-ts": 1909}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1906 (Wed Jun 20 06:01:29 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1907 (Wed Jun 20 06:01:29 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "cd751df4", "local_conf_timestamp": 1907, "host-ts": 1906}, "global_maintenance": true} lago.ssh: DEBUG: start task:26e4358e-f727-442a-8fbd-36993514b9d5:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:26e4358e-f727-442a-8fbd-36993514b9d5:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running ef740360 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ef740360 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command ef740360 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1919 (Wed Jun 20 06:01:42 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1919 (Wed Jun 20 06:01:42 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c3b9909b", "local_conf_timestamp": 1919, "host-ts": 1919}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1916 (Wed Jun 20 06:01:39 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1916 (Wed Jun 20 06:01:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a80e35a7", "local_conf_timestamp": 1916, "host-ts": 1916}, "global_maintenance": true} lago.ssh: DEBUG: start task:4a3243ab-6ccd-4be3-a655-8798bab86bb0:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:4a3243ab-6ccd-4be3-a655-8798bab86bb0:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running f1c1b338 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f1c1b338 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command f1c1b338 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1919 (Wed Jun 20 06:01:42 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1919 (Wed Jun 20 06:01:42 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c3b9909b", "local_conf_timestamp": 1919, "host-ts": 1919}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1916 (Wed Jun 20 06:01:39 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1916 (Wed Jun 20 06:01:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a80e35a7", "local_conf_timestamp": 1916, "host-ts": 1916}, "global_maintenance": true} lago.ssh: DEBUG: start task:c20f8341-7a76-406e-8282-6017f466986e:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:c20f8341-7a76-406e-8282-6017f466986e:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running f41ee826 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f41ee826 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command f41ee826 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1919 (Wed Jun 20 06:01:42 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1919 (Wed Jun 20 06:01:42 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c3b9909b", "local_conf_timestamp": 1919, "host-ts": 1919}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1916 (Wed Jun 20 06:01:39 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1916 (Wed Jun 20 06:01:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a80e35a7", "local_conf_timestamp": 1916, "host-ts": 1916}, "global_maintenance": true} lago.ssh: DEBUG: start task:8c8ce52d-b1eb-4286-9c5a-29d8c463d969:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:8c8ce52d-b1eb-4286-9c5a-29d8c463d969:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running f67bcf08 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f67bcf08 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command f67bcf08 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1929 (Wed Jun 20 06:01:52 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1930 (Wed Jun 20 06:01:53 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9639a2ad", "local_conf_timestamp": 1930, "host-ts": 1929}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1926 (Wed Jun 20 06:01:48 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1926 (Wed Jun 20 06:01:49 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9e8a1ce8", "local_conf_timestamp": 1926, "host-ts": 1926}, "global_maintenance": true} lago.ssh: DEBUG: start task:2c53d7d1-716e-4702-9fe3-d023a0b7f9a5:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:2c53d7d1-716e-4702-9fe3-d023a0b7f9a5:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running f8ebef20 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f8ebef20 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command f8ebef20 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1929 (Wed Jun 20 06:01:52 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1930 (Wed Jun 20 06:01:53 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9639a2ad", "local_conf_timestamp": 1930, "host-ts": 1929}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1926 (Wed Jun 20 06:01:48 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1926 (Wed Jun 20 06:01:49 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9e8a1ce8", "local_conf_timestamp": 1926, "host-ts": 1926}, "global_maintenance": true} lago.ssh: DEBUG: start task:1d686ea2-3f31-4bc9-95b6-5da4b56391b5:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:1d686ea2-3f31-4bc9-95b6-5da4b56391b5:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running fbd6f1e4 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command fbd6f1e4 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command fbd6f1e4 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1939 (Wed Jun 20 06:02:02 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1940 (Wed Jun 20 06:02:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7feae0c2", "local_conf_timestamp": 1940, "host-ts": 1939}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1936 (Wed Jun 20 06:01:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1937 (Wed Jun 20 06:02:00 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fbf134bb", "local_conf_timestamp": 1937, "host-ts": 1936}, "global_maintenance": true} lago.ssh: DEBUG: start task:ae7e23ab-0f2f-4fff-bab0-9dac5441a270:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:ae7e23ab-0f2f-4fff-bab0-9dac5441a270:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running fe5aeca4 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command fe5aeca4 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command fe5aeca4 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1939 (Wed Jun 20 06:02:02 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1940 (Wed Jun 20 06:02:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7feae0c2", "local_conf_timestamp": 1940, "host-ts": 1939}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1936 (Wed Jun 20 06:01:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1937 (Wed Jun 20 06:02:00 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fbf134bb", "local_conf_timestamp": 1937, "host-ts": 1936}, "global_maintenance": true} lago.ssh: DEBUG: start task:b2c02eb6-7b73-4b62-92de-7a1ebce44d9b:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:b2c02eb6-7b73-4b62-92de-7a1ebce44d9b:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 00a5f2ce on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 00a5f2ce on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 00a5f2ce on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1939 (Wed Jun 20 06:02:02 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1940 (Wed Jun 20 06:02:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7feae0c2", "local_conf_timestamp": 1940, "host-ts": 1939}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1936 (Wed Jun 20 06:01:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1937 (Wed Jun 20 06:02:00 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fbf134bb", "local_conf_timestamp": 1937, "host-ts": 1936}, "global_maintenance": true} lago.ssh: DEBUG: start task:4e59618c-2adf-4859-ab52-e3a1fd81bff7:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:4e59618c-2adf-4859-ab52-e3a1fd81bff7:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 033e46a8 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 033e46a8 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 033e46a8 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1949 (Wed Jun 20 06:02:12 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1950 (Wed Jun 20 06:02:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "56c40d1f", "local_conf_timestamp": 1950, "host-ts": 1949}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1946 (Wed Jun 20 06:02:09 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1946 (Wed Jun 20 06:02:09 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f3824e76", "local_conf_timestamp": 1946, "host-ts": 1946}, "global_maintenance": true} lago.ssh: DEBUG: start task:4e06eb5d-93a6-43f2-86b5-aa7486225a68:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:4e06eb5d-93a6-43f2-86b5-aa7486225a68:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 05830c0a on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 05830c0a on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 05830c0a on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1949 (Wed Jun 20 06:02:12 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1950 (Wed Jun 20 06:02:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "56c40d1f", "local_conf_timestamp": 1950, "host-ts": 1949}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1946 (Wed Jun 20 06:02:09 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1946 (Wed Jun 20 06:02:09 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f3824e76", "local_conf_timestamp": 1946, "host-ts": 1946}, "global_maintenance": true} lago.ssh: DEBUG: start task:e0bec1c4-518a-4a4c-b0b5-2f3193d32a7d:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:e0bec1c4-518a-4a4c-b0b5-2f3193d32a7d:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 07deb7ec on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 07deb7ec on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 07deb7ec on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1959 (Wed Jun 20 06:02:22 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=1960 (Wed Jun 20 06:02:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "66b6e371", "local_conf_timestamp": 1960, "host-ts": 1959}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1956 (Wed Jun 20 06:02:18 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1956 (Wed Jun 20 06:02:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e1fe56b3", "local_conf_timestamp": 1956, "host-ts": 1956}, "global_maintenance": true} lago.ssh: DEBUG: start task:8edf7531-89c1-4504-9efc-6c58912e1004:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:8edf7531-89c1-4504-9efc-6c58912e1004:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 0a321da4 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 0a321da4 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 0a321da4 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1959 (Wed Jun 20 06:02:22 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=1960 (Wed Jun 20 06:02:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "66b6e371", "local_conf_timestamp": 1960, "host-ts": 1959}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1956 (Wed Jun 20 06:02:18 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1956 (Wed Jun 20 06:02:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "en ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f578d303398> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 139, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 83, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) RuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1959 (Wed Jun 20 06:02:22 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=1960 (Wed Jun 20 06:02:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "66b6e371", "local_conf_timestamp": 1960, "host-ts": 1959}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1956 (Wed Jun 20 06:02:18 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1956 (Wed Jun 20 06:02:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "en --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 47, in restart_he_vm _wait_for_engine_health(host) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 137, in _wait_for_engine_health testlib.assert_true_within_long(lambda: any( File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 271, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 258, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 139, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 83, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) 'could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1959 (Wed Jun 20 06:02:22 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=1960 (Wed Jun 20 06:02:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "66b6e371", "local_conf_timestamp": 1960, "host-ts": 1959}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1956 (Wed Jun 20 06:02:18 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1956 (Wed Jun 20 06:02:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "en\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:7fa0b63b-1d82-4f26-b3b0-43a232ee42f2:Get ssh client for lago-he-basic-suite-4-2-host-1:\nlago.ssh: DEBUG: end task:7fa0b63b-1d82-4f26-b3b0-43a232ee42f2:Get ssh client for lago-he-basic-suite-4-2-host-1:\nlago.ssh: DEBUG: Running a8297fbc on lago-he-basic-suite-4-2-host-1: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a8297fbc on lago-he-basic-suite-4-2-host-1 returned with 0\nlago.ssh: DEBUG: Command a8297fbc on lago-he-basic-suite-4-2-host-1 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1797 (Wed Jun 20 05:59:40 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1797 (Wed Jun 20 05:59:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8ca7d5c5", "local_conf_timestamp": 1797, "host-ts": 1797}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1796 (Wed Jun 20 05:59:39 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1797 (Wed Jun 20 05:59:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6e153d34", "local_conf_timestamp": 1797, "host-ts": 1796}, "global_maintenance": true}\n\nroot: INFO: * Shutting down HE VM on host: lago-he-basic-suite-4-2-host-0\nlago.ssh: DEBUG: start task:5a3e26ae-7a8e-4aa7-936d-05bac937aebb:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:5a3e26ae-7a8e-4aa7-936d-05bac937aebb:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running a8b75fd0 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-shutdown\nlago.ssh: DEBUG: Command a8b75fd0 on lago-he-basic-suite-4-2-host-0 returned with 0\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be down...\nlago.ssh: DEBUG: start task:8798cb39-f09b-4e2e-bc19-6e4b12083bd5:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:8798cb39-f09b-4e2e-bc19-6e4b12083bd5:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running a9444742 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a9444742 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command a9444742 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1797 (Wed Jun 20 05:59:40 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1797 (Wed Jun 20 05:59:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8ca7d5c5", "local_conf_timestamp": 1797, "host-ts": 1797}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1796 (Wed Jun 20 05:59:39 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1797 (Wed Jun 20 05:59:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6e153d34", "local_conf_timestamp": 1797, "host-ts": 1796}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:18301a7b-63cf-479c-a7cd-abc56f300691:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:18301a7b-63cf-479c-a7cd-abc56f300691:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b00ee8e8 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b00ee8e8 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command b00ee8e8 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1807 (Wed Jun 20 05:59:51 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1809 (Wed Jun 20 05:59:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e73d4537", "local_conf_timestamp": 1809, "host-ts": 1807}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1806 (Wed Jun 20 05:59:49 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1806 (Wed Jun 20 05:59:49 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "516e6745", "local_conf_timestamp": 1806, "host-ts": 1806}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:74ea2f58-62fe-48d7-879f-ccaa247d0b69:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:74ea2f58-62fe-48d7-879f-ccaa247d0b69:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b276bc8c on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b276bc8c on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command b276bc8c on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1817 (Wed Jun 20 06:00:00 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=1817 (Wed Jun 20 06:00:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e573deaf", "local_conf_timestamp": 1817, "host-ts": 1817}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1816 (Wed Jun 20 05:59:58 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1816 (Wed Jun 20 05:59:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "43127f80", "local_conf_timestamp": 1816, "host-ts": 1816}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:eafbaa04-76f1-4346-bbf2-f5154b162e05:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:eafbaa04-76f1-4346-bbf2-f5154b162e05:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b4c932bc on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b4c932bc on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command b4c932bc on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1817 (Wed Jun 20 06:00:00 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=1817 (Wed Jun 20 06:00:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e573deaf", "local_conf_timestamp": 1817, "host-ts": 1817}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1816 (Wed Jun 20 05:59:58 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1816 (Wed Jun 20 05:59:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "43127f80", "local_conf_timestamp": 1816, "host-ts": 1816}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:7e37932e-79d8-4fb1-b018-e509c1be2079:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:7e37932e-79d8-4fb1-b018-e509c1be2079:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b71e9f16 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b71e9f16 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command b71e9f16 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1817 (Wed Jun 20 06:00:00 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=1817 (Wed Jun 20 06:00:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e573deaf", "local_conf_timestamp": 1817, "host-ts": 1817}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1816 (Wed Jun 20 05:59:58 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1816 (Wed Jun 20 05:59:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "43127f80", "local_conf_timestamp": 1816, "host-ts": 1816}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:6721a1b8-c0be-4589-aa6f-06d4593145d7:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:6721a1b8-c0be-4589-aa6f-06d4593145d7:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b97a0c64 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b97a0c64 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command b97a0c64 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1827 (Wed Jun 20 06:00:10 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1827 (Wed Jun 20 06:00:10 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "82f1748b", "local_conf_timestamp": 1827, "host-ts": 1827}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1826 (Wed Jun 20 06:00:09 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1827 (Wed Jun 20 06:00:09 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "02916659", "local_conf_timestamp": 1827, "host-ts": 1826}, "global_maintenance": true}\n\nroot: INFO: * VM is down.\nroot: INFO: * Restarting services...\nlago.ssh: DEBUG: start task:f277519c-ff1c-4145-b766-1529193f9736:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:f277519c-ff1c-4145-b766-1529193f9736:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running ba107a50 on lago-he-basic-suite-4-2-host-0: systemctl restart vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command ba107a50 on lago-he-basic-suite-4-2-host-0 returned with 0\nroot: INFO: * Success.\nroot: INFO: * Waiting for agent to be ready...\nlago.ssh: DEBUG: start task:4720df71-76bb-44f1-b960-8d29c77e7eeb:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:4720df71-76bb-44f1-b960-8d29c77e7eeb:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running c5a97c9a on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command c5a97c9a on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command c5a97c9a on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:31ff44a2-deb7-4f53-af62-ada0d3de7814:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:31ff44a2-deb7-4f53-af62-ada0d3de7814:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running cc151d64 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command cc151d64 on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command cc151d64 on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:dafa50c4-5878-45d5-baf2-f29137211116:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:dafa50c4-5878-45d5-baf2-f29137211116:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running ce3e3648 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command ce3e3648 on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command ce3e3648 on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:c67b6f95-f884-4671-ab08-58e01a0e0ada:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:c67b6f95-f884-4671-ab08-58e01a0e0ada:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running d082176c on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command d082176c on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command d082176c on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:1a9235d2-a03f-445b-b0c0-bc2e74f552a2:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:1a9235d2-a03f-445b-b0c0-bc2e74f552a2:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running d2aad948 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command d2aad948 on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command d2aad948 on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:6b6ea68c-dd38-4096-a0ac-dd38a55e2688:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:6b6ea68c-dd38-4096-a0ac-dd38a55e2688:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running d4d59258 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command d4d59258 on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command d4d59258 on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:b5a78ba3-a0fa-4cdf-ac2d-f1ebd5340516:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:b5a78ba3-a0fa-4cdf-ac2d-f1ebd5340516:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running d710521a on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command d710521a on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command d710521a on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:d74eb21b-e2d0-48af-95a6-6ae0d9c322b2:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:d74eb21b-e2d0-48af-95a6-6ae0d9c322b2:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running d93d74f0 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command d93d74f0 on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command d93d74f0 on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:d5883737-42de-4bd9-b14c-3aa39c8f2dfd:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:d5883737-42de-4bd9-b14c-3aa39c8f2dfd:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running db7573b2 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command db7573b2 on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command db7573b2 on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:695e26b3-2326-47f0-81e6-8ba946e0eab9:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:695e26b3-2326-47f0-81e6-8ba946e0eab9:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running dd92f5b6 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command dd92f5b6 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command dd92f5b6 on lago-he-basic-suite-4-2-host-0 output:\n \n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\n\n--== Host 1 status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : True\nHostname : lago-he-basic-suite-4-2-host-0.lago.local\nHost ID : 1\nEngine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}\nScore : 0\nstopped : False\nLocal maintenance : False\ncrc32 : 612645ed\nlocal_conf_timestamp : 1889\nHost timestamp : 1889\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=1889 (Wed Jun 20 06:01:12 2018)\n\thost-id=1\n\tscore=0\n\tvm_conf_refresh_time=1889 (Wed Jun 20 06:01:13 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=ReinitializeFSM\n\tstopped=False\n\n\n--== Host 2 status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : False\nHostname : lago-he-basic-suite-4-2-host-1\nHost ID : 2\nEngine status : unknown stale-data\nScore : 3400\nstopped : False\nLocal maintenance : False\ncrc32 : c28ea16d\nlocal_conf_timestamp : 1886\nHost timestamp : 1886\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=1886 (Wed Jun 20 06:01:09 2018)\n\thost-id=2\n\tscore=3400\n\tvm_conf_refresh_time=1886 (Wed Jun 20 06:01:09 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=GlobalMaintenance\n\tstopped=False\n\n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\nroot: INFO: * Agent is ready.\nroot: INFO: * Starting VM...\nlago.ssh: DEBUG: start task:34e67d6d-becb-4934-b953-d59e64f2a55c:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:34e67d6d-becb-4934-b953-d59e64f2a55c:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running de2286cc on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-start\nlago.ssh: DEBUG: Command de2286cc on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command de2286cc on lago-he-basic-suite-4-2-host-0 output:\n VM exists and is down, cleaning up and restarting\n\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting to VM to be UP...\nlago.ssh: DEBUG: start task:aa25364d-1a02-4256-82d9-50f4799364b3:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:aa25364d-1a02-4256-82d9-50f4799364b3:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running df8a2718 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command df8a2718 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command df8a2718 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1889 (Wed Jun 20 06:01:12 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=1889 (Wed Jun 20 06:01:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=ReinitializeFSM\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "612645ed", "local_conf_timestamp": 1889, "host-ts": 1889}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1886 (Wed Jun 20 06:01:09 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1886 (Wed Jun 20 06:01:09 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c28ea16d", "local_conf_timestamp": 1886, "host-ts": 1886}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:807fa695-4172-4455-961d-5aab46991388:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:807fa695-4172-4455-961d-5aab46991388:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running e611a3a4 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e611a3a4 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command e611a3a4 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1899 (Wed Jun 20 06:01:22 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1899 (Wed Jun 20 06:01:22 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e627649f", "local_conf_timestamp": 1899, "host-ts": 1899}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1896 (Wed Jun 20 06:01:18 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1896 (Wed Jun 20 06:01:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d0f2b9a8", "local_conf_timestamp": 1896, "host-ts": 1896}, "global_maintenance": true}\n\nroot: INFO: * VM is UP.\nroot: INFO: * Waiting for engine to start ...\nlago.ssh: DEBUG: start task:dd69f784-ffa7-437e-8146-4b4550768f6f:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:dd69f784-ffa7-437e-8146-4b4550768f6f:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running e69adf3e on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e69adf3e on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command e69adf3e on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1899 (Wed Jun 20 06:01:22 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1899 (Wed Jun 20 06:01:22 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e627649f", "local_conf_timestamp": 1899, "host-ts": 1899}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1896 (Wed Jun 20 06:01:18 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1896 (Wed Jun 20 06:01:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d0f2b9a8", "local_conf_timestamp": 1896, "host-ts": 1896}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:73996669-0bfe-4f86-a81d-ee17c02c5b7c:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:73996669-0bfe-4f86-a81d-ee17c02c5b7c:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running ed261ac6 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ed261ac6 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command ed261ac6 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1909 (Wed Jun 20 06:01:32 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1910 (Wed Jun 20 06:01:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "06938df0", "local_conf_timestamp": 1910, "host-ts": 1909}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1906 (Wed Jun 20 06:01:29 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1907 (Wed Jun 20 06:01:29 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "cd751df4", "local_conf_timestamp": 1907, "host-ts": 1906}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:26e4358e-f727-442a-8fbd-36993514b9d5:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:26e4358e-f727-442a-8fbd-36993514b9d5:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running ef740360 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ef740360 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command ef740360 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1919 (Wed Jun 20 06:01:42 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1919 (Wed Jun 20 06:01:42 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c3b9909b", "local_conf_timestamp": 1919, "host-ts": 1919}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1916 (Wed Jun 20 06:01:39 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1916 (Wed Jun 20 06:01:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a80e35a7", "local_conf_timestamp": 1916, "host-ts": 1916}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:4a3243ab-6ccd-4be3-a655-8798bab86bb0:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:4a3243ab-6ccd-4be3-a655-8798bab86bb0:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running f1c1b338 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f1c1b338 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command f1c1b338 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1919 (Wed Jun 20 06:01:42 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1919 (Wed Jun 20 06:01:42 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c3b9909b", "local_conf_timestamp": 1919, "host-ts": 1919}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1916 (Wed Jun 20 06:01:39 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1916 (Wed Jun 20 06:01:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a80e35a7", "local_conf_timestamp": 1916, "host-ts": 1916}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:c20f8341-7a76-406e-8282-6017f466986e:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:c20f8341-7a76-406e-8282-6017f466986e:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running f41ee826 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f41ee826 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command f41ee826 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1919 (Wed Jun 20 06:01:42 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1919 (Wed Jun 20 06:01:42 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c3b9909b", "local_conf_timestamp": 1919, "host-ts": 1919}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1916 (Wed Jun 20 06:01:39 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1916 (Wed Jun 20 06:01:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a80e35a7", "local_conf_timestamp": 1916, "host-ts": 1916}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8c8ce52d-b1eb-4286-9c5a-29d8c463d969:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:8c8ce52d-b1eb-4286-9c5a-29d8c463d969:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running f67bcf08 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f67bcf08 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command f67bcf08 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1929 (Wed Jun 20 06:01:52 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1930 (Wed Jun 20 06:01:53 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9639a2ad", "local_conf_timestamp": 1930, "host-ts": 1929}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1926 (Wed Jun 20 06:01:48 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1926 (Wed Jun 20 06:01:49 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9e8a1ce8", "local_conf_timestamp": 1926, "host-ts": 1926}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:2c53d7d1-716e-4702-9fe3-d023a0b7f9a5:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:2c53d7d1-716e-4702-9fe3-d023a0b7f9a5:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running f8ebef20 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f8ebef20 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command f8ebef20 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1929 (Wed Jun 20 06:01:52 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1930 (Wed Jun 20 06:01:53 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9639a2ad", "local_conf_timestamp": 1930, "host-ts": 1929}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1926 (Wed Jun 20 06:01:48 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1926 (Wed Jun 20 06:01:49 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9e8a1ce8", "local_conf_timestamp": 1926, "host-ts": 1926}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:1d686ea2-3f31-4bc9-95b6-5da4b56391b5:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:1d686ea2-3f31-4bc9-95b6-5da4b56391b5:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running fbd6f1e4 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fbd6f1e4 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command fbd6f1e4 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1939 (Wed Jun 20 06:02:02 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1940 (Wed Jun 20 06:02:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7feae0c2", "local_conf_timestamp": 1940, "host-ts": 1939}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1936 (Wed Jun 20 06:01:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1937 (Wed Jun 20 06:02:00 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fbf134bb", "local_conf_timestamp": 1937, "host-ts": 1936}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ae7e23ab-0f2f-4fff-bab0-9dac5441a270:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:ae7e23ab-0f2f-4fff-bab0-9dac5441a270:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running fe5aeca4 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fe5aeca4 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command fe5aeca4 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1939 (Wed Jun 20 06:02:02 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1940 (Wed Jun 20 06:02:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7feae0c2", "local_conf_timestamp": 1940, "host-ts": 1939}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1936 (Wed Jun 20 06:01:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1937 (Wed Jun 20 06:02:00 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fbf134bb", "local_conf_timestamp": 1937, "host-ts": 1936}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:b2c02eb6-7b73-4b62-92de-7a1ebce44d9b:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:b2c02eb6-7b73-4b62-92de-7a1ebce44d9b:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 00a5f2ce on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 00a5f2ce on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 00a5f2ce on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1939 (Wed Jun 20 06:02:02 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1940 (Wed Jun 20 06:02:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7feae0c2", "local_conf_timestamp": 1940, "host-ts": 1939}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1936 (Wed Jun 20 06:01:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1937 (Wed Jun 20 06:02:00 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fbf134bb", "local_conf_timestamp": 1937, "host-ts": 1936}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:4e59618c-2adf-4859-ab52-e3a1fd81bff7:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:4e59618c-2adf-4859-ab52-e3a1fd81bff7:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 033e46a8 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 033e46a8 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 033e46a8 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1949 (Wed Jun 20 06:02:12 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1950 (Wed Jun 20 06:02:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "56c40d1f", "local_conf_timestamp": 1950, "host-ts": 1949}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1946 (Wed Jun 20 06:02:09 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1946 (Wed Jun 20 06:02:09 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f3824e76", "local_conf_timestamp": 1946, "host-ts": 1946}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:4e06eb5d-93a6-43f2-86b5-aa7486225a68:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:4e06eb5d-93a6-43f2-86b5-aa7486225a68:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 05830c0a on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 05830c0a on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 05830c0a on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1949 (Wed Jun 20 06:02:12 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1950 (Wed Jun 20 06:02:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "56c40d1f", "local_conf_timestamp": 1950, "host-ts": 1949}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1946 (Wed Jun 20 06:02:09 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1946 (Wed Jun 20 06:02:09 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f3824e76", "local_conf_timestamp": 1946, "host-ts": 1946}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:e0bec1c4-518a-4a4c-b0b5-2f3193d32a7d:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:e0bec1c4-518a-4a4c-b0b5-2f3193d32a7d:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 07deb7ec on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 07deb7ec on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 07deb7ec on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1959 (Wed Jun 20 06:02:22 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=1960 (Wed Jun 20 06:02:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "66b6e371", "local_conf_timestamp": 1960, "host-ts": 1959}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1956 (Wed Jun 20 06:02:18 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1956 (Wed Jun 20 06:02:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e1fe56b3", "local_conf_timestamp": 1956, "host-ts": 1956}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8edf7531-89c1-4504-9efc-6c58912e1004:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:8edf7531-89c1-4504-9efc-6c58912e1004:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 0a321da4 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 0a321da4 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 0a321da4 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1959 (Wed Jun 20 06:02:22 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=1960 (Wed Jun 20 06:02:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "66b6e371", "local_conf_timestamp": 1960, "host-ts": 1959}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1956 (Wed Jun 20 06:02:18 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1956 (Wed Jun 20 06:02:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "en\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f578d303398>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 139, in <lambda>\n for k, v in _get_he_status(host).items()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 83, in _get_he_status\n raise RuntimeError(\'could not parse JSON: %s\' % ret.out)\nRuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1959 (Wed Jun 20 06:02:22 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=1960 (Wed Jun 20 06:02:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "66b6e371", "local_conf_timestamp": 1960, "host-ts": 1959}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1956 (Wed Jun 20 06:02:18 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1956 (Wed Jun 20 06:02:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "en\n--------------------- >> end captured logging << ---------------------'

1 1

[CQ]: 92124, 15 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 20 Jun '18

20 Jun '18

A system test invoked by the "ovirt-master" change queue including change 92124,15 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 92305,5 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 92305,5 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/92124/15 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/92305/5 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8301/

1 0

[CQ]: 92271, 11 (ovirt-hosted-engine-setup) failed "ovirt-master" system tests
by oVirt Jenkins 20 Jun '18

20 Jun '18

Change 92271,11 (ovirt-hosted-engine-setup) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/92271/11 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8300/

1 0

[CQ]: 92082, 1 (ovirt-engine-sdk-ruby) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 20 Jun '18

20 Jun '18

A system test invoked by the "ovirt-master" change queue including change 92082,1 (ovirt-engine-sdk-ruby) failed. However, this change seems not to be the root cause for this failure. Change 89845,1 (ovirt-engine-sdk-ruby) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 89845,1 (ovirt-engine-sdk-ruby) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/92082/1 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/89845/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8298/

1 0

[CQ]: 92305,5 (ovirt-engine) failed "ovirt-master" system tests
by oVirt Jenkins 20 Jun '18

20 Jun '18

Change 92305,5 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/92305/5 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8296/

2 1

[JIRA] (OVIRT-2225) 001_upgrade_engine.test_initialize_engine failed due to missing ovirt-host-deploy pacakge
by Dafna Ron (oVirt JIRA) 20 Jun '18

20 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2225?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2225: ---------------------------------- ovirt-host-deploy is provided by python2-ovirt-host-deploy https://gerrit.ovirt.org/92405 > 001_upgrade_engine.test_initialize_engine failed due to missing ovirt-host-deploy pacakge > ----------------------------------------------------------------------------------------- > > Key: OVIRT-2225 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2225 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_infra, ost_infra_packages > > The error we are failing on is: > ("FATAL Internal error (main): [u'ovirt-host-deploy-java-1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch requires ovirt-host-deploy = 1.8.0-0.0.master.20180620111438.git925eabd.el7']",) > Looking at the reposync ovirt-host-deploy-* have a new version except for ovirt-host-deploy itself. > 'Repository google-chrome is listed more than once in the configuration > Repository google-talkplugin is listed more than once in the configuration > ovirt-host-deploy-0:1.8.0-0.0.master.20180531090832.git9811a30.el7.noarch > ovirt-host-deploy-common-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch > ovirt-host-deploy-java-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch > ovirt-host-deploy-javadoc-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch > python2-ovirt-host-deploy-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch > [dron@dron ovirt-system-tests]$ ' > I ran the reposync file to see if the location changed but it did not. I also excluded the package from tested to see it its available somewhere else but its not. > asking Sandro to take a look. > Here are the failed change and build: > For further details about the change see: > https://gerrit.ovirt.org/#/c/92381/1 > For failed test results see: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8293/ -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-2225) 001_upgrade_engine.test_initialize_engine failed due to missing ovirt-host-deploy pacakge
by Dafna Ron (oVirt JIRA) 20 Jun '18

20 Jun '18

Dafna Ron created OVIRT-2225: -------------------------------- Summary: 001_upgrade_engine.test_initialize_engine failed due to missing ovirt-host-deploy pacakge Key: OVIRT-2225 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2225 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Dafna Ron Assignee: infra The error we are failing on is: ("FATAL Internal error (main): [u'ovirt-host-deploy-java-1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch requires ovirt-host-deploy = 1.8.0-0.0.master.20180620111438.git925eabd.el7']",) Looking at the reposync ovirt-host-deploy-* have a new version except for ovirt-host-deploy itself. 'Repository google-chrome is listed more than once in the configuration Repository google-talkplugin is listed more than once in the configuration ovirt-host-deploy-0:1.8.0-0.0.master.20180531090832.git9811a30.el7.noarch ovirt-host-deploy-common-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch ovirt-host-deploy-java-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch ovirt-host-deploy-javadoc-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch python2-ovirt-host-deploy-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch [dron@dron ovirt-system-tests]$ ' I ran the reposync file to see if the location changed but it did not. I also excluded the package from tested to see it its available somewhere else but its not. asking Sandro to take a look. Here are the failed change and build: For further details about the change see: https://gerrit.ovirt.org/#/c/92381/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8293/ -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build # 496 - Failure!
by jenkins＠jenkins.phx.ovirt.org 20 Jun '18

20 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/496/ Build Number: 496 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #496 [Gal Ben Haim] upgrade_from_release_suite_master Update pre-reposync ----------------- Failed Tests: ----------------- No tests ran.

1 2