June 2018 - Infra - oVirt List Archives

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #1679
by jenkins＠jenkins.phx.ovirt.org 21 Jun '18

21 Jun '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 7e8a988cfa3a5cde022712ab4142923ebffbb4f4 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 7e8a988cfa3a5cde022712ab4142923ebffbb4f4 Commit message: "Move get_generic_queue_build_args to stdci_project" > git rev-list --no-walk 7e8a988cfa3a5cde022712ab4142923ebffbb4f4 # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins8105867571467064723.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7' Build step 'Execute shell' marked build as failure

1 1

[JIRA] (OVIRT-2225) 001_upgrade_engine.test_initialize_engine failed due to missing ovirt-host-deploy pacakge
by Barak Korren (oVirt JIRA) 21 Jun '18

21 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2225?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-2225: ------------------------------------- so basically this is an ovirt-engine regression and the specfile needx to be fixed? > 001_upgrade_engine.test_initialize_engine failed due to missing ovirt-host-deploy pacakge > ----------------------------------------------------------------------------------------- > > Key: OVIRT-2225 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2225 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_infra, ost_infra_packages > > The error we are failing on is: > ("FATAL Internal error (main): [u'ovirt-host-deploy-java-1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch requires ovirt-host-deploy = 1.8.0-0.0.master.20180620111438.git925eabd.el7']",) > Looking at the reposync ovirt-host-deploy-* have a new version except for ovirt-host-deploy itself. > 'Repository google-chrome is listed more than once in the configuration > Repository google-talkplugin is listed more than once in the configuration > ovirt-host-deploy-0:1.8.0-0.0.master.20180531090832.git9811a30.el7.noarch > ovirt-host-deploy-common-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch > ovirt-host-deploy-java-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch > ovirt-host-deploy-javadoc-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch > python2-ovirt-host-deploy-0:1.8.0-0.0.master.20180620111438.git925eabd.el7.noarch > [dron@dron ovirt-system-tests]$ ' > I ran the reposync file to see if the location changed but it did not. I also excluded the package from tested to see it its available somewhere else but its not. > asking Sandro to take a look. > Here are the failed change and build: > For further details about the change see: > https://gerrit.ovirt.org/#/c/92381/1 > For failed test results see: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8293/ -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[oVirt Jenkins] ovirt-system-tests_he-basic-iscsi-suite-master - Build # 262 - Failure!
by jenkins＠jenkins.phx.ovirt.org 21 Jun '18

21 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master… Build Number: 262 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #262 [Sandro Bonazzola] yum-repos: master: adjusted for new packages [Daniel Belenky] Move get_generic_queue_build_args to stdci_project ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.add_secondary_storage_domains Error Message: Fault reason is "Operation Failed". Fault detail is "[Cannot add storage server connection when Host status is not up]". HTTP response code is 409. -------------------- >> begin captured logging << -------------------- lago.utils: ERROR: Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 491, in add_nfs_storage_domain add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, SD_NFS_PATH) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 496, in add_generic_nfs_storage_domain add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 552, in add_generic_nfs_storage_domain_4 _add_storage_domain_4(api, p) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 447, in _add_storage_domain_4 sd = sds_service.add(p) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 24011, in add return self._internal_add(storage_domain, headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault self._raise_error(response, body) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error Error: Fault reason is "Operation Failed". Fault detail is "[Cannot add storage server connection when Host status is not up]". HTTP response code is 409. --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 577, in add_secondary_storage_domains vt.join_all() File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 491, in add_nfs_storage_domain add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, SD_NFS_PATH) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 496, in add_generic_nfs_storage_domain add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 552, in add_generic_nfs_storage_domain_4 _add_storage_domain_4(api, p) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 447, in _add_storage_domain_4 sd = sds_service.add(p) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 24011, in add return self._internal_add(storage_domain, headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault self._raise_error(response, body) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error 'Fault reason is "Operation Failed". Fault detail is "[Cannot add storage server connection when Host status is not up]". HTTP response code is 409.\n-------------------- >> begin captured logging << --------------------\nlago.utils: ERROR: Error while running thread\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue\n queue.put({\'return\': func()})\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 491, in add_nfs_storage_domain\n add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, SD_NFS_PATH)\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 496, in add_generic_nfs_storage_domain\n add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 552, in add_generic_nfs_storage_domain_4\n _add_storage_domain_4(api, p)\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/002_bootstrap.py", line 447, in _add_storage_domain_4\n sd = sds_service.add(p)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 24011, in add\n return self._internal_add(storage_domain, headers, query, wait)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add\n return future.wait() if wait else future\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait\n return self._code(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback\n self._check_fault(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault\n self._raise_error(response, body)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error\n raise error\nError: Fault reason is "Operation Failed". Fault detail is "[Cannot add storage server connection when Host status is not up]". HTTP response code is 409.\n--------------------- >> end captured logging << ---------------------'

1 1

[oVirt Jenkins] ovirt-system-tests_network-suite-master - Build # 319 - Failure!
by jenkins＠jenkins.phx.ovirt.org 21 Jun '18

21 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-master/319/ Build Number: 319 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #319 [Milan Zamazal] Test snapshot with memory in 4.2 [Daniel Belenky] Move get_generic_queue_build_args to stdci_project ----------------- Failed Tests: ----------------- No tests ran.

1 1

[oVirt Jenkins] ovirt-system-tests_performance-suite-master - Build # 419 - Failure!
by jenkins＠jenkins.phx.ovirt.org 21 Jun '18

21 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_performance-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_performance-suite-master/41… Build Number: 419 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #419 [Milan Zamazal] Test snapshot with memory in 4.2 [Daniel Belenky] Move get_generic_queue_build_args to stdci_project ----------------- Failed Tests: ----------------- No tests ran.

1 1

[oVirt Jenkins] ovirt-system-tests_he-basic-ansible-suite-master - Build # 372 - Failure!
by jenkins＠jenkins.phx.ovirt.org 21 Jun '18

21 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-mast… Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-mast… Build Number: 372 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #372 [Gal Ben Haim] upgrade_from_release_suite_master Update pre-reposync [Daniel Belenky] Install python-six during global-setup [Barak Korren] Make mock_runner use curl instead of wget [Barak Korren] Fix proxied configuration for FC27 and FC28 ----------------- Failed Tests: ----------------- All tests passed

1 2

oVirt infra weekly meeting notes 20.06.2018
by Evgheni Dereveanchin 20 Jun '18

20 Jun '18

Hi everyone, Please find the topics of this week's infra meeting below: *PHX - FC26 slaves replaced with FC28- GID issue with mock OVIRT-2210- python-six issue OVIRT-2218- dnf-makecache patch ready for merge OVIRT-1927- CVE patching - ticket opened to patch el7 slaves- Puppet check-patch migration to STDCI re-started- Monitoring migration - access to PHX open form new box OVIRT-502- CentOS bare metal sharing - port access requested OVIRT-2179* -- Regards, Evgheni Dereveanchin

1 0

[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-master - Build # 118 - Failure!
by jenkins＠jenkins.phx.ovirt.org 20 Jun '18

20 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/118/ Build Number: 118 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #118 [Milan Zamazal] Test snapshot with memory in 4.2 [Daniel Belenky] Install python-six during global-setup [Barak Korren] Make mock_runner use curl instead of wget [Barak Korren] Fix proxied configuration for FC27 and FC28 [Daniel Belenky] Move project functions to a shared lib [Daniel Belenky] Utilize stdci_project groovy lib in CQ [Daniel Belenky] Move get_generic_queue_build_args to stdci_project ----------------- Failed Tests: ----------------- All tests passed

1 1

[oVirt Jenkins] ovirt-system-tests_he-basic-suite-4.2 - Build # 307 - Failure!
by jenkins＠jenkins.phx.ovirt.org 20 Jun '18

20 Jun '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.2/307/ Build Number: 307 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #307 [Gal Ben Haim] upgrade_from_release_suite_master Update pre-reposync [Daniel Belenky] Install python-six during global-setup [Barak Korren] Make mock_runner use curl instead of wget [Barak Korren] Fix proxied configuration for FC27 and FC28 [Daniel Belenky] Move project functions to a shared lib [Daniel Belenky] Utilize stdci_project groovy lib in CQ ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 008_restart_he_vm.restart_he_vm Error Message: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1959 (Wed Jun 20 06:02:22 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=1960 (Wed Jun 20 06:02:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "66b6e371", "local_conf_timestamp": 1960, "host-ts": 1959}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1956 (Wed Jun 20 06:02:18 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1956 (Wed Jun 20 06:02:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "en -------------------- >> begin captured logging << -------------------- lago.ssh: DEBUG: start task:7fa0b63b-1d82-4f26-b3b0-43a232ee42f2:Get ssh client for lago-he-basic-suite-4-2-host-1: lago.ssh: DEBUG: end task:7fa0b63b-1d82-4f26-b3b0-43a232ee42f2:Get ssh client for lago-he-basic-suite-4-2-host-1: lago.ssh: DEBUG: Running a8297fbc on lago-he-basic-suite-4-2-host-1: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a8297fbc on lago-he-basic-suite-4-2-host-1 returned with 0 lago.ssh: DEBUG: Command a8297fbc on lago-he-basic-suite-4-2-host-1 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1797 (Wed Jun 20 05:59:40 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1797 (Wed Jun 20 05:59:40 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8ca7d5c5", "local_conf_timestamp": 1797, "host-ts": 1797}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1796 (Wed Jun 20 05:59:39 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1797 (Wed Jun 20 05:59:40 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6e153d34", "local_conf_timestamp": 1797, "host-ts": 1796}, "global_maintenance": true} root: INFO: * Shutting down HE VM on host: lago-he-basic-suite-4-2-host-0 lago.ssh: DEBUG: start task:5a3e26ae-7a8e-4aa7-936d-05bac937aebb:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:5a3e26ae-7a8e-4aa7-936d-05bac937aebb:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running a8b75fd0 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-shutdown lago.ssh: DEBUG: Command a8b75fd0 on lago-he-basic-suite-4-2-host-0 returned with 0 root: INFO: * Command succeeded root: INFO: * Waiting for VM to be down... lago.ssh: DEBUG: start task:8798cb39-f09b-4e2e-bc19-6e4b12083bd5:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:8798cb39-f09b-4e2e-bc19-6e4b12083bd5:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running a9444742 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a9444742 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command a9444742 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1797 (Wed Jun 20 05:59:40 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1797 (Wed Jun 20 05:59:40 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8ca7d5c5", "local_conf_timestamp": 1797, "host-ts": 1797}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1796 (Wed Jun 20 05:59:39 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1797 (Wed Jun 20 05:59:40 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6e153d34", "local_conf_timestamp": 1797, "host-ts": 1796}, "global_maintenance": false} lago.ssh: DEBUG: start task:18301a7b-63cf-479c-a7cd-abc56f300691:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:18301a7b-63cf-479c-a7cd-abc56f300691:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running b00ee8e8 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b00ee8e8 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command b00ee8e8 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1807 (Wed Jun 20 05:59:51 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1809 (Wed Jun 20 05:59:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e73d4537", "local_conf_timestamp": 1809, "host-ts": 1807}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1806 (Wed Jun 20 05:59:49 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1806 (Wed Jun 20 05:59:49 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "516e6745", "local_conf_timestamp": 1806, "host-ts": 1806}, "global_maintenance": true} lago.ssh: DEBUG: start task:74ea2f58-62fe-48d7-879f-ccaa247d0b69:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:74ea2f58-62fe-48d7-879f-ccaa247d0b69:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running b276bc8c on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b276bc8c on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command b276bc8c on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1817 (Wed Jun 20 06:00:00 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=1817 (Wed Jun 20 06:00:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e573deaf", "local_conf_timestamp": 1817, "host-ts": 1817}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1816 (Wed Jun 20 05:59:58 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1816 (Wed Jun 20 05:59:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "43127f80", "local_conf_timestamp": 1816, "host-ts": 1816}, "global_maintenance": true} lago.ssh: DEBUG: start task:eafbaa04-76f1-4346-bbf2-f5154b162e05:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:eafbaa04-76f1-4346-bbf2-f5154b162e05:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running b4c932bc on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b4c932bc on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command b4c932bc on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1817 (Wed Jun 20 06:00:00 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=1817 (Wed Jun 20 06:00:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e573deaf", "local_conf_timestamp": 1817, "host-ts": 1817}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1816 (Wed Jun 20 05:59:58 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1816 (Wed Jun 20 05:59:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "43127f80", "local_conf_timestamp": 1816, "host-ts": 1816}, "global_maintenance": true} lago.ssh: DEBUG: start task:7e37932e-79d8-4fb1-b018-e509c1be2079:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:7e37932e-79d8-4fb1-b018-e509c1be2079:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running b71e9f16 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b71e9f16 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command b71e9f16 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1817 (Wed Jun 20 06:00:00 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=1817 (Wed Jun 20 06:00:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e573deaf", "local_conf_timestamp": 1817, "host-ts": 1817}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1816 (Wed Jun 20 05:59:58 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1816 (Wed Jun 20 05:59:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "43127f80", "local_conf_timestamp": 1816, "host-ts": 1816}, "global_maintenance": true} lago.ssh: DEBUG: start task:6721a1b8-c0be-4589-aa6f-06d4593145d7:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:6721a1b8-c0be-4589-aa6f-06d4593145d7:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running b97a0c64 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b97a0c64 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command b97a0c64 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1827 (Wed Jun 20 06:00:10 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1827 (Wed Jun 20 06:00:10 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "82f1748b", "local_conf_timestamp": 1827, "host-ts": 1827}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1826 (Wed Jun 20 06:00:09 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1827 (Wed Jun 20 06:00:09 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "02916659", "local_conf_timestamp": 1827, "host-ts": 1826}, "global_maintenance": true} root: INFO: * VM is down. root: INFO: * Restarting services... lago.ssh: DEBUG: start task:f277519c-ff1c-4145-b766-1529193f9736:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:f277519c-ff1c-4145-b766-1529193f9736:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running ba107a50 on lago-he-basic-suite-4-2-host-0: systemctl restart vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command ba107a50 on lago-he-basic-suite-4-2-host-0 returned with 0 root: INFO: * Success. root: INFO: * Waiting for agent to be ready... lago.ssh: DEBUG: start task:4720df71-76bb-44f1-b960-8d29c77e7eeb:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:4720df71-76bb-44f1-b960-8d29c77e7eeb:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running c5a97c9a on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command c5a97c9a on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command c5a97c9a on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:31ff44a2-deb7-4f53-af62-ada0d3de7814:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:31ff44a2-deb7-4f53-af62-ada0d3de7814:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running cc151d64 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command cc151d64 on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command cc151d64 on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:dafa50c4-5878-45d5-baf2-f29137211116:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:dafa50c4-5878-45d5-baf2-f29137211116:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running ce3e3648 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command ce3e3648 on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command ce3e3648 on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:c67b6f95-f884-4671-ab08-58e01a0e0ada:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:c67b6f95-f884-4671-ab08-58e01a0e0ada:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running d082176c on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command d082176c on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command d082176c on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:1a9235d2-a03f-445b-b0c0-bc2e74f552a2:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:1a9235d2-a03f-445b-b0c0-bc2e74f552a2:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running d2aad948 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command d2aad948 on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command d2aad948 on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:6b6ea68c-dd38-4096-a0ac-dd38a55e2688:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:6b6ea68c-dd38-4096-a0ac-dd38a55e2688:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running d4d59258 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command d4d59258 on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command d4d59258 on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:b5a78ba3-a0fa-4cdf-ac2d-f1ebd5340516:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:b5a78ba3-a0fa-4cdf-ac2d-f1ebd5340516:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running d710521a on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command d710521a on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command d710521a on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:d74eb21b-e2d0-48af-95a6-6ae0d9c322b2:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:d74eb21b-e2d0-48af-95a6-6ae0d9c322b2:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running d93d74f0 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command d93d74f0 on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command d93d74f0 on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:d5883737-42de-4bd9-b14c-3aa39c8f2dfd:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:d5883737-42de-4bd9-b14c-3aa39c8f2dfd:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running db7573b2 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command db7573b2 on lago-he-basic-suite-4-2-host-0 returned with 1 lago.ssh: DEBUG: Command db7573b2 on lago-he-basic-suite-4-2-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:695e26b3-2326-47f0-81e6-8ba946e0eab9:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:695e26b3-2326-47f0-81e6-8ba946e0eab9:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running dd92f5b6 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command dd92f5b6 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command dd92f5b6 on lago-he-basic-suite-4-2-host-0 output: !! Cluster is in GLOBAL MAINTENANCE mode !! --== Host 1 status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : lago-he-basic-suite-4-2-host-0.lago.local Host ID : 1 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"} Score : 0 stopped : False Local maintenance : False crc32 : 612645ed local_conf_timestamp : 1889 Host timestamp : 1889 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=1889 (Wed Jun 20 06:01:12 2018) host-id=1 score=0 vm_conf_refresh_time=1889 (Wed Jun 20 06:01:13 2018) conf_on_shared_storage=True maintenance=False state=ReinitializeFSM stopped=False --== Host 2 status ==-- conf_on_shared_storage : True Status up-to-date : False Hostname : lago-he-basic-suite-4-2-host-1 Host ID : 2 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : c28ea16d local_conf_timestamp : 1886 Host timestamp : 1886 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=1886 (Wed Jun 20 06:01:09 2018) host-id=2 score=3400 vm_conf_refresh_time=1886 (Wed Jun 20 06:01:09 2018) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False !! Cluster is in GLOBAL MAINTENANCE mode !! root: INFO: * Agent is ready. root: INFO: * Starting VM... lago.ssh: DEBUG: start task:34e67d6d-becb-4934-b953-d59e64f2a55c:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:34e67d6d-becb-4934-b953-d59e64f2a55c:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running de2286cc on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-start lago.ssh: DEBUG: Command de2286cc on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command de2286cc on lago-he-basic-suite-4-2-host-0 output: VM exists and is down, cleaning up and restarting root: INFO: * Command succeeded root: INFO: * Waiting to VM to be UP... lago.ssh: DEBUG: start task:aa25364d-1a02-4256-82d9-50f4799364b3:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:aa25364d-1a02-4256-82d9-50f4799364b3:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running df8a2718 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command df8a2718 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command df8a2718 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1889 (Wed Jun 20 06:01:12 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=1889 (Wed Jun 20 06:01:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=ReinitializeFSM\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "612645ed", "local_conf_timestamp": 1889, "host-ts": 1889}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1886 (Wed Jun 20 06:01:09 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1886 (Wed Jun 20 06:01:09 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c28ea16d", "local_conf_timestamp": 1886, "host-ts": 1886}, "global_maintenance": true} lago.ssh: DEBUG: start task:807fa695-4172-4455-961d-5aab46991388:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:807fa695-4172-4455-961d-5aab46991388:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running e611a3a4 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e611a3a4 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command e611a3a4 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1899 (Wed Jun 20 06:01:22 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1899 (Wed Jun 20 06:01:22 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e627649f", "local_conf_timestamp": 1899, "host-ts": 1899}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1896 (Wed Jun 20 06:01:18 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1896 (Wed Jun 20 06:01:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d0f2b9a8", "local_conf_timestamp": 1896, "host-ts": 1896}, "global_maintenance": true} root: INFO: * VM is UP. root: INFO: * Waiting for engine to start ... lago.ssh: DEBUG: start task:dd69f784-ffa7-437e-8146-4b4550768f6f:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:dd69f784-ffa7-437e-8146-4b4550768f6f:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running e69adf3e on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command e69adf3e on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command e69adf3e on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1899 (Wed Jun 20 06:01:22 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1899 (Wed Jun 20 06:01:22 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e627649f", "local_conf_timestamp": 1899, "host-ts": 1899}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1896 (Wed Jun 20 06:01:18 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1896 (Wed Jun 20 06:01:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d0f2b9a8", "local_conf_timestamp": 1896, "host-ts": 1896}, "global_maintenance": true} lago.ssh: DEBUG: start task:73996669-0bfe-4f86-a81d-ee17c02c5b7c:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:73996669-0bfe-4f86-a81d-ee17c02c5b7c:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running ed261ac6 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ed261ac6 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command ed261ac6 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1909 (Wed Jun 20 06:01:32 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1910 (Wed Jun 20 06:01:33 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "06938df0", "local_conf_timestamp": 1910, "host-ts": 1909}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1906 (Wed Jun 20 06:01:29 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1907 (Wed Jun 20 06:01:29 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "cd751df4", "local_conf_timestamp": 1907, "host-ts": 1906}, "global_maintenance": true} lago.ssh: DEBUG: start task:26e4358e-f727-442a-8fbd-36993514b9d5:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:26e4358e-f727-442a-8fbd-36993514b9d5:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running ef740360 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ef740360 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command ef740360 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1919 (Wed Jun 20 06:01:42 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1919 (Wed Jun 20 06:01:42 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c3b9909b", "local_conf_timestamp": 1919, "host-ts": 1919}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1916 (Wed Jun 20 06:01:39 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1916 (Wed Jun 20 06:01:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a80e35a7", "local_conf_timestamp": 1916, "host-ts": 1916}, "global_maintenance": true} lago.ssh: DEBUG: start task:4a3243ab-6ccd-4be3-a655-8798bab86bb0:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:4a3243ab-6ccd-4be3-a655-8798bab86bb0:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running f1c1b338 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f1c1b338 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command f1c1b338 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1919 (Wed Jun 20 06:01:42 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1919 (Wed Jun 20 06:01:42 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c3b9909b", "local_conf_timestamp": 1919, "host-ts": 1919}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1916 (Wed Jun 20 06:01:39 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1916 (Wed Jun 20 06:01:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a80e35a7", "local_conf_timestamp": 1916, "host-ts": 1916}, "global_maintenance": true} lago.ssh: DEBUG: start task:c20f8341-7a76-406e-8282-6017f466986e:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:c20f8341-7a76-406e-8282-6017f466986e:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running f41ee826 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f41ee826 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command f41ee826 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1919 (Wed Jun 20 06:01:42 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1919 (Wed Jun 20 06:01:42 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c3b9909b", "local_conf_timestamp": 1919, "host-ts": 1919}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1916 (Wed Jun 20 06:01:39 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1916 (Wed Jun 20 06:01:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a80e35a7", "local_conf_timestamp": 1916, "host-ts": 1916}, "global_maintenance": true} lago.ssh: DEBUG: start task:8c8ce52d-b1eb-4286-9c5a-29d8c463d969:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:8c8ce52d-b1eb-4286-9c5a-29d8c463d969:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running f67bcf08 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f67bcf08 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command f67bcf08 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1929 (Wed Jun 20 06:01:52 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1930 (Wed Jun 20 06:01:53 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9639a2ad", "local_conf_timestamp": 1930, "host-ts": 1929}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1926 (Wed Jun 20 06:01:48 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1926 (Wed Jun 20 06:01:49 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9e8a1ce8", "local_conf_timestamp": 1926, "host-ts": 1926}, "global_maintenance": true} lago.ssh: DEBUG: start task:2c53d7d1-716e-4702-9fe3-d023a0b7f9a5:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:2c53d7d1-716e-4702-9fe3-d023a0b7f9a5:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running f8ebef20 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command f8ebef20 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command f8ebef20 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1929 (Wed Jun 20 06:01:52 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1930 (Wed Jun 20 06:01:53 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9639a2ad", "local_conf_timestamp": 1930, "host-ts": 1929}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1926 (Wed Jun 20 06:01:48 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1926 (Wed Jun 20 06:01:49 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9e8a1ce8", "local_conf_timestamp": 1926, "host-ts": 1926}, "global_maintenance": true} lago.ssh: DEBUG: start task:1d686ea2-3f31-4bc9-95b6-5da4b56391b5:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:1d686ea2-3f31-4bc9-95b6-5da4b56391b5:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running fbd6f1e4 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command fbd6f1e4 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command fbd6f1e4 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1939 (Wed Jun 20 06:02:02 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1940 (Wed Jun 20 06:02:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7feae0c2", "local_conf_timestamp": 1940, "host-ts": 1939}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1936 (Wed Jun 20 06:01:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1937 (Wed Jun 20 06:02:00 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fbf134bb", "local_conf_timestamp": 1937, "host-ts": 1936}, "global_maintenance": true} lago.ssh: DEBUG: start task:ae7e23ab-0f2f-4fff-bab0-9dac5441a270:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:ae7e23ab-0f2f-4fff-bab0-9dac5441a270:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running fe5aeca4 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command fe5aeca4 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command fe5aeca4 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1939 (Wed Jun 20 06:02:02 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1940 (Wed Jun 20 06:02:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7feae0c2", "local_conf_timestamp": 1940, "host-ts": 1939}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1936 (Wed Jun 20 06:01:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1937 (Wed Jun 20 06:02:00 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fbf134bb", "local_conf_timestamp": 1937, "host-ts": 1936}, "global_maintenance": true} lago.ssh: DEBUG: start task:b2c02eb6-7b73-4b62-92de-7a1ebce44d9b:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:b2c02eb6-7b73-4b62-92de-7a1ebce44d9b:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 00a5f2ce on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 00a5f2ce on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 00a5f2ce on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1939 (Wed Jun 20 06:02:02 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1940 (Wed Jun 20 06:02:03 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7feae0c2", "local_conf_timestamp": 1940, "host-ts": 1939}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1936 (Wed Jun 20 06:01:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1937 (Wed Jun 20 06:02:00 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fbf134bb", "local_conf_timestamp": 1937, "host-ts": 1936}, "global_maintenance": true} lago.ssh: DEBUG: start task:4e59618c-2adf-4859-ab52-e3a1fd81bff7:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:4e59618c-2adf-4859-ab52-e3a1fd81bff7:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 033e46a8 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 033e46a8 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 033e46a8 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1949 (Wed Jun 20 06:02:12 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1950 (Wed Jun 20 06:02:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "56c40d1f", "local_conf_timestamp": 1950, "host-ts": 1949}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1946 (Wed Jun 20 06:02:09 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1946 (Wed Jun 20 06:02:09 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f3824e76", "local_conf_timestamp": 1946, "host-ts": 1946}, "global_maintenance": true} lago.ssh: DEBUG: start task:4e06eb5d-93a6-43f2-86b5-aa7486225a68:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:4e06eb5d-93a6-43f2-86b5-aa7486225a68:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 05830c0a on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 05830c0a on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 05830c0a on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1949 (Wed Jun 20 06:02:12 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=1950 (Wed Jun 20 06:02:13 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "56c40d1f", "local_conf_timestamp": 1950, "host-ts": 1949}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1946 (Wed Jun 20 06:02:09 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1946 (Wed Jun 20 06:02:09 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f3824e76", "local_conf_timestamp": 1946, "host-ts": 1946}, "global_maintenance": true} lago.ssh: DEBUG: start task:e0bec1c4-518a-4a4c-b0b5-2f3193d32a7d:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:e0bec1c4-518a-4a4c-b0b5-2f3193d32a7d:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 07deb7ec on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 07deb7ec on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 07deb7ec on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1959 (Wed Jun 20 06:02:22 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=1960 (Wed Jun 20 06:02:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "66b6e371", "local_conf_timestamp": 1960, "host-ts": 1959}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1956 (Wed Jun 20 06:02:18 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1956 (Wed Jun 20 06:02:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e1fe56b3", "local_conf_timestamp": 1956, "host-ts": 1956}, "global_maintenance": true} lago.ssh: DEBUG: start task:8edf7531-89c1-4504-9efc-6c58912e1004:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: end task:8edf7531-89c1-4504-9efc-6c58912e1004:Get ssh client for lago-he-basic-suite-4-2-host-0: lago.ssh: DEBUG: Running 0a321da4 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 0a321da4 on lago-he-basic-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 0a321da4 on lago-he-basic-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1959 (Wed Jun 20 06:02:22 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=1960 (Wed Jun 20 06:02:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "66b6e371", "local_conf_timestamp": 1960, "host-ts": 1959}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1956 (Wed Jun 20 06:02:18 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1956 (Wed Jun 20 06:02:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "en ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f578d303398> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 139, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 83, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) RuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1959 (Wed Jun 20 06:02:22 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=1960 (Wed Jun 20 06:02:23 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "66b6e371", "local_conf_timestamp": 1960, "host-ts": 1959}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1956 (Wed Jun 20 06:02:18 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1956 (Wed Jun 20 06:02:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "en --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 47, in restart_he_vm _wait_for_engine_health(host) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 137, in _wait_for_engine_health testlib.assert_true_within_long(lambda: any( File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 271, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 258, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 139, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 83, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) 'could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1959 (Wed Jun 20 06:02:22 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=1960 (Wed Jun 20 06:02:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "66b6e371", "local_conf_timestamp": 1960, "host-ts": 1959}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1956 (Wed Jun 20 06:02:18 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1956 (Wed Jun 20 06:02:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "en\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:7fa0b63b-1d82-4f26-b3b0-43a232ee42f2:Get ssh client for lago-he-basic-suite-4-2-host-1:\nlago.ssh: DEBUG: end task:7fa0b63b-1d82-4f26-b3b0-43a232ee42f2:Get ssh client for lago-he-basic-suite-4-2-host-1:\nlago.ssh: DEBUG: Running a8297fbc on lago-he-basic-suite-4-2-host-1: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a8297fbc on lago-he-basic-suite-4-2-host-1 returned with 0\nlago.ssh: DEBUG: Command a8297fbc on lago-he-basic-suite-4-2-host-1 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1797 (Wed Jun 20 05:59:40 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1797 (Wed Jun 20 05:59:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8ca7d5c5", "local_conf_timestamp": 1797, "host-ts": 1797}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1796 (Wed Jun 20 05:59:39 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1797 (Wed Jun 20 05:59:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6e153d34", "local_conf_timestamp": 1797, "host-ts": 1796}, "global_maintenance": true}\n\nroot: INFO: * Shutting down HE VM on host: lago-he-basic-suite-4-2-host-0\nlago.ssh: DEBUG: start task:5a3e26ae-7a8e-4aa7-936d-05bac937aebb:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:5a3e26ae-7a8e-4aa7-936d-05bac937aebb:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running a8b75fd0 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-shutdown\nlago.ssh: DEBUG: Command a8b75fd0 on lago-he-basic-suite-4-2-host-0 returned with 0\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be down...\nlago.ssh: DEBUG: start task:8798cb39-f09b-4e2e-bc19-6e4b12083bd5:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:8798cb39-f09b-4e2e-bc19-6e4b12083bd5:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running a9444742 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a9444742 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command a9444742 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1797 (Wed Jun 20 05:59:40 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1797 (Wed Jun 20 05:59:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8ca7d5c5", "local_conf_timestamp": 1797, "host-ts": 1797}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1796 (Wed Jun 20 05:59:39 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1797 (Wed Jun 20 05:59:40 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6e153d34", "local_conf_timestamp": 1797, "host-ts": 1796}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:18301a7b-63cf-479c-a7cd-abc56f300691:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:18301a7b-63cf-479c-a7cd-abc56f300691:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b00ee8e8 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b00ee8e8 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command b00ee8e8 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1807 (Wed Jun 20 05:59:51 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1809 (Wed Jun 20 05:59:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e73d4537", "local_conf_timestamp": 1809, "host-ts": 1807}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1806 (Wed Jun 20 05:59:49 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1806 (Wed Jun 20 05:59:49 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "516e6745", "local_conf_timestamp": 1806, "host-ts": 1806}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:74ea2f58-62fe-48d7-879f-ccaa247d0b69:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:74ea2f58-62fe-48d7-879f-ccaa247d0b69:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b276bc8c on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b276bc8c on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command b276bc8c on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1817 (Wed Jun 20 06:00:00 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=1817 (Wed Jun 20 06:00:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e573deaf", "local_conf_timestamp": 1817, "host-ts": 1817}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1816 (Wed Jun 20 05:59:58 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1816 (Wed Jun 20 05:59:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "43127f80", "local_conf_timestamp": 1816, "host-ts": 1816}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:eafbaa04-76f1-4346-bbf2-f5154b162e05:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:eafbaa04-76f1-4346-bbf2-f5154b162e05:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b4c932bc on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b4c932bc on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command b4c932bc on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1817 (Wed Jun 20 06:00:00 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=1817 (Wed Jun 20 06:00:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e573deaf", "local_conf_timestamp": 1817, "host-ts": 1817}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1816 (Wed Jun 20 05:59:58 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1816 (Wed Jun 20 05:59:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "43127f80", "local_conf_timestamp": 1816, "host-ts": 1816}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:7e37932e-79d8-4fb1-b018-e509c1be2079:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:7e37932e-79d8-4fb1-b018-e509c1be2079:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b71e9f16 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b71e9f16 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command b71e9f16 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1817 (Wed Jun 20 06:00:00 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=1817 (Wed Jun 20 06:00:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "e573deaf", "local_conf_timestamp": 1817, "host-ts": 1817}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1816 (Wed Jun 20 05:59:58 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1816 (Wed Jun 20 05:59:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "43127f80", "local_conf_timestamp": 1816, "host-ts": 1816}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:6721a1b8-c0be-4589-aa6f-06d4593145d7:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:6721a1b8-c0be-4589-aa6f-06d4593145d7:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b97a0c64 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b97a0c64 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command b97a0c64 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1827 (Wed Jun 20 06:00:10 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1827 (Wed Jun 20 06:00:10 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "82f1748b", "local_conf_timestamp": 1827, "host-ts": 1827}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1826 (Wed Jun 20 06:00:09 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1827 (Wed Jun 20 06:00:09 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "02916659", "local_conf_timestamp": 1827, "host-ts": 1826}, "global_maintenance": true}\n\nroot: INFO: * VM is down.\nroot: INFO: * Restarting services...\nlago.ssh: DEBUG: start task:f277519c-ff1c-4145-b766-1529193f9736:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:f277519c-ff1c-4145-b766-1529193f9736:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running ba107a50 on lago-he-basic-suite-4-2-host-0: systemctl restart vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command ba107a50 on lago-he-basic-suite-4-2-host-0 returned with 0\nroot: INFO: * Success.\nroot: INFO: * Waiting for agent to be ready...\nlago.ssh: DEBUG: start task:4720df71-76bb-44f1-b960-8d29c77e7eeb:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:4720df71-76bb-44f1-b960-8d29c77e7eeb:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running c5a97c9a on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command c5a97c9a on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command c5a97c9a on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:31ff44a2-deb7-4f53-af62-ada0d3de7814:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:31ff44a2-deb7-4f53-af62-ada0d3de7814:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running cc151d64 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command cc151d64 on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command cc151d64 on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:dafa50c4-5878-45d5-baf2-f29137211116:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:dafa50c4-5878-45d5-baf2-f29137211116:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running ce3e3648 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command ce3e3648 on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command ce3e3648 on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:c67b6f95-f884-4671-ab08-58e01a0e0ada:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:c67b6f95-f884-4671-ab08-58e01a0e0ada:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running d082176c on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command d082176c on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command d082176c on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:1a9235d2-a03f-445b-b0c0-bc2e74f552a2:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:1a9235d2-a03f-445b-b0c0-bc2e74f552a2:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running d2aad948 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command d2aad948 on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command d2aad948 on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:6b6ea68c-dd38-4096-a0ac-dd38a55e2688:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:6b6ea68c-dd38-4096-a0ac-dd38a55e2688:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running d4d59258 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command d4d59258 on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command d4d59258 on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:b5a78ba3-a0fa-4cdf-ac2d-f1ebd5340516:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:b5a78ba3-a0fa-4cdf-ac2d-f1ebd5340516:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running d710521a on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command d710521a on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command d710521a on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:d74eb21b-e2d0-48af-95a6-6ae0d9c322b2:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:d74eb21b-e2d0-48af-95a6-6ae0d9c322b2:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running d93d74f0 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command d93d74f0 on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command d93d74f0 on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:d5883737-42de-4bd9-b14c-3aa39c8f2dfd:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:d5883737-42de-4bd9-b14c-3aa39c8f2dfd:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running db7573b2 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command db7573b2 on lago-he-basic-suite-4-2-host-0 returned with 1\nlago.ssh: DEBUG: Command db7573b2 on lago-he-basic-suite-4-2-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:695e26b3-2326-47f0-81e6-8ba946e0eab9:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:695e26b3-2326-47f0-81e6-8ba946e0eab9:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running dd92f5b6 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command dd92f5b6 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command dd92f5b6 on lago-he-basic-suite-4-2-host-0 output:\n \n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\n\n--== Host 1 status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : True\nHostname : lago-he-basic-suite-4-2-host-0.lago.local\nHost ID : 1\nEngine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}\nScore : 0\nstopped : False\nLocal maintenance : False\ncrc32 : 612645ed\nlocal_conf_timestamp : 1889\nHost timestamp : 1889\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=1889 (Wed Jun 20 06:01:12 2018)\n\thost-id=1\n\tscore=0\n\tvm_conf_refresh_time=1889 (Wed Jun 20 06:01:13 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=ReinitializeFSM\n\tstopped=False\n\n\n--== Host 2 status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : False\nHostname : lago-he-basic-suite-4-2-host-1\nHost ID : 2\nEngine status : unknown stale-data\nScore : 3400\nstopped : False\nLocal maintenance : False\ncrc32 : c28ea16d\nlocal_conf_timestamp : 1886\nHost timestamp : 1886\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=1886 (Wed Jun 20 06:01:09 2018)\n\thost-id=2\n\tscore=3400\n\tvm_conf_refresh_time=1886 (Wed Jun 20 06:01:09 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=GlobalMaintenance\n\tstopped=False\n\n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\nroot: INFO: * Agent is ready.\nroot: INFO: * Starting VM...\nlago.ssh: DEBUG: start task:34e67d6d-becb-4934-b953-d59e64f2a55c:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:34e67d6d-becb-4934-b953-d59e64f2a55c:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running de2286cc on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-start\nlago.ssh: DEBUG: Command de2286cc on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command de2286cc on lago-he-basic-suite-4-2-host-0 output:\n VM exists and is down, cleaning up and restarting\n\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting to VM to be UP...\nlago.ssh: DEBUG: start task:aa25364d-1a02-4256-82d9-50f4799364b3:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:aa25364d-1a02-4256-82d9-50f4799364b3:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running df8a2718 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command df8a2718 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command df8a2718 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1889 (Wed Jun 20 06:01:12 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=1889 (Wed Jun 20 06:01:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=ReinitializeFSM\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "612645ed", "local_conf_timestamp": 1889, "host-ts": 1889}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1886 (Wed Jun 20 06:01:09 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1886 (Wed Jun 20 06:01:09 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c28ea16d", "local_conf_timestamp": 1886, "host-ts": 1886}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:807fa695-4172-4455-961d-5aab46991388:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:807fa695-4172-4455-961d-5aab46991388:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running e611a3a4 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e611a3a4 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command e611a3a4 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1899 (Wed Jun 20 06:01:22 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1899 (Wed Jun 20 06:01:22 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e627649f", "local_conf_timestamp": 1899, "host-ts": 1899}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1896 (Wed Jun 20 06:01:18 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1896 (Wed Jun 20 06:01:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d0f2b9a8", "local_conf_timestamp": 1896, "host-ts": 1896}, "global_maintenance": true}\n\nroot: INFO: * VM is UP.\nroot: INFO: * Waiting for engine to start ...\nlago.ssh: DEBUG: start task:dd69f784-ffa7-437e-8146-4b4550768f6f:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:dd69f784-ffa7-437e-8146-4b4550768f6f:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running e69adf3e on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command e69adf3e on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command e69adf3e on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1899 (Wed Jun 20 06:01:22 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1899 (Wed Jun 20 06:01:22 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e627649f", "local_conf_timestamp": 1899, "host-ts": 1899}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1896 (Wed Jun 20 06:01:18 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1896 (Wed Jun 20 06:01:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d0f2b9a8", "local_conf_timestamp": 1896, "host-ts": 1896}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:73996669-0bfe-4f86-a81d-ee17c02c5b7c:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:73996669-0bfe-4f86-a81d-ee17c02c5b7c:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running ed261ac6 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ed261ac6 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command ed261ac6 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1909 (Wed Jun 20 06:01:32 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1910 (Wed Jun 20 06:01:33 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "06938df0", "local_conf_timestamp": 1910, "host-ts": 1909}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1906 (Wed Jun 20 06:01:29 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1907 (Wed Jun 20 06:01:29 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "cd751df4", "local_conf_timestamp": 1907, "host-ts": 1906}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:26e4358e-f727-442a-8fbd-36993514b9d5:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:26e4358e-f727-442a-8fbd-36993514b9d5:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running ef740360 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ef740360 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command ef740360 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1919 (Wed Jun 20 06:01:42 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1919 (Wed Jun 20 06:01:42 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c3b9909b", "local_conf_timestamp": 1919, "host-ts": 1919}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1916 (Wed Jun 20 06:01:39 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1916 (Wed Jun 20 06:01:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a80e35a7", "local_conf_timestamp": 1916, "host-ts": 1916}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:4a3243ab-6ccd-4be3-a655-8798bab86bb0:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:4a3243ab-6ccd-4be3-a655-8798bab86bb0:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running f1c1b338 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f1c1b338 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command f1c1b338 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1919 (Wed Jun 20 06:01:42 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1919 (Wed Jun 20 06:01:42 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c3b9909b", "local_conf_timestamp": 1919, "host-ts": 1919}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1916 (Wed Jun 20 06:01:39 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1916 (Wed Jun 20 06:01:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a80e35a7", "local_conf_timestamp": 1916, "host-ts": 1916}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:c20f8341-7a76-406e-8282-6017f466986e:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:c20f8341-7a76-406e-8282-6017f466986e:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running f41ee826 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f41ee826 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command f41ee826 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1919 (Wed Jun 20 06:01:42 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1919 (Wed Jun 20 06:01:42 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c3b9909b", "local_conf_timestamp": 1919, "host-ts": 1919}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1916 (Wed Jun 20 06:01:39 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1916 (Wed Jun 20 06:01:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a80e35a7", "local_conf_timestamp": 1916, "host-ts": 1916}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8c8ce52d-b1eb-4286-9c5a-29d8c463d969:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:8c8ce52d-b1eb-4286-9c5a-29d8c463d969:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running f67bcf08 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f67bcf08 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command f67bcf08 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1929 (Wed Jun 20 06:01:52 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1930 (Wed Jun 20 06:01:53 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9639a2ad", "local_conf_timestamp": 1930, "host-ts": 1929}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1926 (Wed Jun 20 06:01:48 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1926 (Wed Jun 20 06:01:49 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9e8a1ce8", "local_conf_timestamp": 1926, "host-ts": 1926}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:2c53d7d1-716e-4702-9fe3-d023a0b7f9a5:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:2c53d7d1-716e-4702-9fe3-d023a0b7f9a5:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running f8ebef20 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command f8ebef20 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command f8ebef20 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1929 (Wed Jun 20 06:01:52 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1930 (Wed Jun 20 06:01:53 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9639a2ad", "local_conf_timestamp": 1930, "host-ts": 1929}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1926 (Wed Jun 20 06:01:48 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1926 (Wed Jun 20 06:01:49 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9e8a1ce8", "local_conf_timestamp": 1926, "host-ts": 1926}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:1d686ea2-3f31-4bc9-95b6-5da4b56391b5:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:1d686ea2-3f31-4bc9-95b6-5da4b56391b5:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running fbd6f1e4 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fbd6f1e4 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command fbd6f1e4 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1939 (Wed Jun 20 06:02:02 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1940 (Wed Jun 20 06:02:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7feae0c2", "local_conf_timestamp": 1940, "host-ts": 1939}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1936 (Wed Jun 20 06:01:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1937 (Wed Jun 20 06:02:00 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fbf134bb", "local_conf_timestamp": 1937, "host-ts": 1936}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ae7e23ab-0f2f-4fff-bab0-9dac5441a270:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:ae7e23ab-0f2f-4fff-bab0-9dac5441a270:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running fe5aeca4 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command fe5aeca4 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command fe5aeca4 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1939 (Wed Jun 20 06:02:02 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1940 (Wed Jun 20 06:02:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7feae0c2", "local_conf_timestamp": 1940, "host-ts": 1939}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1936 (Wed Jun 20 06:01:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1937 (Wed Jun 20 06:02:00 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fbf134bb", "local_conf_timestamp": 1937, "host-ts": 1936}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:b2c02eb6-7b73-4b62-92de-7a1ebce44d9b:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:b2c02eb6-7b73-4b62-92de-7a1ebce44d9b:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 00a5f2ce on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 00a5f2ce on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 00a5f2ce on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1939 (Wed Jun 20 06:02:02 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1940 (Wed Jun 20 06:02:03 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7feae0c2", "local_conf_timestamp": 1940, "host-ts": 1939}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1936 (Wed Jun 20 06:01:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1937 (Wed Jun 20 06:02:00 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fbf134bb", "local_conf_timestamp": 1937, "host-ts": 1936}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:4e59618c-2adf-4859-ab52-e3a1fd81bff7:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:4e59618c-2adf-4859-ab52-e3a1fd81bff7:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 033e46a8 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 033e46a8 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 033e46a8 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1949 (Wed Jun 20 06:02:12 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1950 (Wed Jun 20 06:02:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "56c40d1f", "local_conf_timestamp": 1950, "host-ts": 1949}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1946 (Wed Jun 20 06:02:09 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1946 (Wed Jun 20 06:02:09 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f3824e76", "local_conf_timestamp": 1946, "host-ts": 1946}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:4e06eb5d-93a6-43f2-86b5-aa7486225a68:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:4e06eb5d-93a6-43f2-86b5-aa7486225a68:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 05830c0a on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 05830c0a on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 05830c0a on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1949 (Wed Jun 20 06:02:12 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=1950 (Wed Jun 20 06:02:13 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "56c40d1f", "local_conf_timestamp": 1950, "host-ts": 1949}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1946 (Wed Jun 20 06:02:09 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1946 (Wed Jun 20 06:02:09 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f3824e76", "local_conf_timestamp": 1946, "host-ts": 1946}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:e0bec1c4-518a-4a4c-b0b5-2f3193d32a7d:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:e0bec1c4-518a-4a4c-b0b5-2f3193d32a7d:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 07deb7ec on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 07deb7ec on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 07deb7ec on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1959 (Wed Jun 20 06:02:22 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=1960 (Wed Jun 20 06:02:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "66b6e371", "local_conf_timestamp": 1960, "host-ts": 1959}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1956 (Wed Jun 20 06:02:18 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1956 (Wed Jun 20 06:02:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e1fe56b3", "local_conf_timestamp": 1956, "host-ts": 1956}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8edf7531-89c1-4504-9efc-6c58912e1004:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:8edf7531-89c1-4504-9efc-6c58912e1004:Get ssh client for lago-he-basic-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 0a321da4 on lago-he-basic-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 0a321da4 on lago-he-basic-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 0a321da4 on lago-he-basic-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1959 (Wed Jun 20 06:02:22 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=1960 (Wed Jun 20 06:02:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "66b6e371", "local_conf_timestamp": 1960, "host-ts": 1959}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1956 (Wed Jun 20 06:02:18 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1956 (Wed Jun 20 06:02:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "en\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f578d303398>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 139, in <lambda>\n for k, v in _get_he_status(host).items()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/008_restart_he_vm.py", line 83, in _get_he_status\n raise RuntimeError(\'could not parse JSON: %s\' % ret.out)\nRuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1959 (Wed Jun 20 06:02:22 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=1960 (Wed Jun 20 06:02:23 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "66b6e371", "local_conf_timestamp": 1960, "host-ts": 1959}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1956 (Wed Jun 20 06:02:18 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=1956 (Wed Jun 20 06:02:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-suite-4-2-host-1", "host-id": 2, "en\n--------------------- >> end captured logging << ---------------------'

1 1

[CQ]: 92124, 15 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 20 Jun '18

20 Jun '18

A system test invoked by the "ovirt-master" change queue including change 92124,15 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 92305,5 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 92305,5 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/92124/15 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/92305/5 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8301/

1 0