August 2018 - Infra - Ovirt List Archives

[oVirt Jenkins] ovirt-node-ng-image_master_build-artifacts-fc28-x86_64 - Build # 34 - Failure!

by jenkins＠jenkins.phx.ovirt.org

Project: http://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-f... Build: http://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-f... Build Number: 34 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #34 [Yuval Turgeman] stdci: move build-artifacts to a v1 job ----------------- Failed Tests: ----------------- No tests ran.

6 years, 2 months

1
2
0 / 0

[JIRA] (OVIRT-2453) vdsm_{master,4.2} check-merge to run on bare metal slaves on CI

by Daniel Belenky (oVirt JIRA)

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2453?page=com.atlassian.jir... ] Daniel Belenky commented on OVIRT-2453: --------------------------------------- On Tue, Aug 28, 2018 at 8:23 AM Edward Haas <ehaas(a)redhat.com> wrote: > Hello, > > VDSM check-merge is using lago VM/s to run functional tests, mainly in the > network domain. > Unfortunately, when running on a VM slave, the tests are running in a > nested VM which seems to be unstable (the nested VM is getting > stuck/crashes). > > Here are some recent failures: > http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/247/ > http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/249/ > > Could you please run this job on a bare metal slave? > > Thanks, > Edy. > -- DANIEL BELENKY > vdsm_{master,4.2} check-merge to run on bare metal slaves on CI > --------------------------------------------------------------- > > Key: OVIRT-2453 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2453 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Edward Haas > Assignee: infra > > Hello, > VDSM check-merge is using lago VM/s to run functional tests, mainly in the > network domain. > Unfortunately, when running on a VM slave, the tests are running in a > nested VM which seems to be unstable (the nested VM is getting > stuck/crashes). > Here are some recent failures: > http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/247/ > http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/249/ > Could you please run this job on a bare metal slave? > Thanks, > Edy. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

6 years, 2 months

1
0
0 / 0

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #1886

by jenkins＠jenkins.phx.ovirt.org

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 8aa2a8538d0bc192415e0eac6c831cbe94d68793 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 8aa2a8538d0bc192415e0eac6c831cbe94d68793 Commit message: "mock_runner: Set no_proxy for proxied configs" > git rev-list --no-walk 8aa2a8538d0bc192415e0eac6c831cbe94d68793 # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins2807930280698784854.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7' Build step 'Execute shell' marked build as failure

6 years, 2 months

1
1
0 / 0

[JIRA] (OVIRT-2453) vdsm_{master,4.2} check-merge to run on bare metal slaves on CI

by Edward Haas (oVirt JIRA)

Edward Haas created OVIRT-2453: ---------------------------------- Summary: vdsm_{master,4.2} check-merge to run on bare metal slaves on CI Key: OVIRT-2453 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2453 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Edward Haas Assignee: infra Hello, VDSM check-merge is using lago VM/s to run functional tests, mainly in the network domain. Unfortunately, when running on a VM slave, the tests are running in a nested VM which seems to be unstable (the nested VM is getting stuck/crashes). Here are some recent failures: http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/247/ http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/249/ Could you please run this job on a bare metal slave? Thanks, Edy. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

6 years, 2 months

1
0
0 / 0

Build failed in Jenkins: system-sync_mirrors-glusterfs-4.1-test-el7-x86_64 #1

by jenkins＠jenkins.phx.ovirt.org

See <http://jenkins.ovirt.org/job/system-sync_mirrors-glusterfs-4.1-test-el7-x...> ------------------------------------------ Started by user Ehud Yonasi [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-glusterfs-4.1-test-el7-x...> Cloning the remote Git repository Cloning repository http://gerrit.ovirt.org/jenkins.git > git init <http://jenkins.ovirt.org/job/system-sync_mirrors-glusterfs-4.1-test-el7-x...> # timeout=10 Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 No valid HEAD. Skipping the resetting > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 8aa2a8538d0bc192415e0eac6c831cbe94d68793 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 8aa2a8538d0bc192415e0eac6c831cbe94d68793 Commit message: "mock_runner: Set no_proxy for proxied configs" First time build. Skipping changelog. [system-sync_mirrors-glusterfs-4.1-test-el7-x86_64] $ /bin/bash -xe /tmp/jenkins5612468679084601662.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror glusterfs-4.1-test-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Warning: cannot find repository glusterfs-4.1-test-el7 No repositories found Build step 'Execute shell' marked build as failure

6 years, 2 months

2
4
0 / 0

oVirt infra daily report - unstable production jobs - 565

by jenkins＠jenkins.phx.ovirt.org

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/565//artifact/exported... Cheers, Jenkins

6 years, 2 months

1
0
0 / 0

[CQ]: 93925,2 (vdsm) failed "ovirt-4.2" system tests

by oVirt Jenkins

Change 93925,2 (vdsm) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/93925/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2993/

6 years, 2 months

1
0
0 / 0

[oVirt Jenkins] ovirt-system-tests_he-basic-iscsi-suite-master - Build # 464 - Failure!

by jenkins＠jenkins.phx.ovirt.org

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-mast... Build Number: 464 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #464 [Roy Golan] Introduce openshift-on-ovirt ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 008_restart_he_vm.restart_he_vm Error Message: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", -------------------- >> begin captured logging << -------------------- lago.ssh: DEBUG: start task:a3c71d1c-cb12-4fef-9e6a-d5e0c1b07550:Get ssh client for lago-he-basic-iscsi-suite-master-host-1: lago.ssh: DEBUG: end task:a3c71d1c-cb12-4fef-9e6a-d5e0c1b07550:Get ssh client for lago-he-basic-iscsi-suite-master-host-1: lago.ssh: DEBUG: Running 43cfc44a on lago-he-basic-iscsi-suite-master-host-1: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 43cfc44a on lago-he-basic-iscsi-suite-master-host-1 returned with 0 lago.ssh: DEBUG: Command 43cfc44a on lago-he-basic-iscsi-suite-master-host-1 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2805 (Sun Aug 26 23:28:20 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2806 (Sun Aug 26 23:28:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd22cf78", "local_conf_timestamp": 2806, "host-ts": 2805}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2810 (Sun Aug 26 23:28:24 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2812 (Sun Aug 26 23:28:26 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7f7e634d", "local_conf_timestamp": 2812, "host-ts": 2810}, "global_maintenance": true} root: INFO: * Shutting down HE VM on host: lago-he-basic-iscsi-suite-master-host-0 lago.ssh: DEBUG: start task:42afe970-0be5-40d6-834b-245d877c0d1a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:42afe970-0be5-40d6-834b-245d877c0d1a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 445276b0 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-shutdown lago.ssh: DEBUG: Command 445276b0 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 root: INFO: * Command succeeded root: INFO: * Waiting for VM to be down... lago.ssh: DEBUG: start task:d92ec1bd-a571-4e86-b3a6-7bedd805567d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:d92ec1bd-a571-4e86-b3a6-7bedd805567d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 44f97960 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 44f97960 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 44f97960 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2805 (Sun Aug 26 23:28:20 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2806 (Sun Aug 26 23:28:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd22cf78", "local_conf_timestamp": 2806, "host-ts": 2805}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2800 (Sun Aug 26 23:28:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2801 (Sun Aug 26 23:28:15 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "063aa920", "local_conf_timestamp": 2801, "host-ts": 2800}, "global_maintenance": false} lago.ssh: DEBUG: start task:a1e22b81-c1cf-4b25-a1f7-342286fbe81b:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:a1e22b81-c1cf-4b25-a1f7-342286fbe81b:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true} lago.ssh: DEBUG: start task:f5948091-c497-4f3b-9761-04d42a2cc421:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:f5948091-c497-4f3b-9761-04d42a2cc421:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 4df162c6 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 4df162c6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 4df162c6 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true} lago.ssh: DEBUG: start task:8b462547-686e-4dbe-8bfb-91d4ba2931dd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:8b462547-686e-4dbe-8bfb-91d4ba2931dd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 504ec626 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 504ec626 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 504ec626 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true} lago.ssh: DEBUG: start task:f2eef89a-f42e-4c79-9337-72571575f679:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:f2eef89a-f42e-4c79-9337-72571575f679:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 52a4916c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 52a4916c on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 52a4916c on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2835 (Sun Aug 26 23:28:50 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2837 (Sun Aug 26 23:28:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7a4710c6", "local_conf_timestamp": 2837, "host-ts": 2835}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2831 (Sun Aug 26 23:28:45 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2833 (Sun Aug 26 23:28:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fce69ba8", "local_conf_timestamp": 2833, "host-ts": 2831}, "global_maintenance": true} root: INFO: * VM is down. root: INFO: * Stopping services... lago.ssh: DEBUG: start task:2c25f4a6-f2ef-4cfa-822f-b929c4dfd509:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:2c25f4a6-f2ef-4cfa-822f-b929c4dfd509:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 5322b5b0 on lago-he-basic-iscsi-suite-master-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command 5322b5b0 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 root: INFO: * Starting services... lago.ssh: DEBUG: start task:ee6b4609-b5ac-4f3c-a182-7ed44b8deeb6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:ee6b4609-b5ac-4f3c-a182-7ed44b8deeb6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 5a5b1dd6 on lago-he-basic-iscsi-suite-master-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command 5a5b1dd6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 root: INFO: * Waiting for agent to be ready... lago.ssh: DEBUG: start task:facd8e55-61ed-4fb7-b9fe-c3ad705d968a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:facd8e55-61ed-4fb7-b9fe-c3ad705d968a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:3eb14b7c-074b-4c11-a117-392f8b162c2e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:3eb14b7c-074b-4c11-a117-392f8b162c2e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 62a124ea on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 62a124ea on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 62a124ea on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:c78b3a53-f7bb-480e-839e-b57e9ca7eb4d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:c78b3a53-f7bb-480e-839e-b57e9ca7eb4d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:ca525d74-b2b6-4297-ad54-b044bc452568:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:ca525d74-b2b6-4297-ad54-b044bc452568:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 66d48ffc on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 66d48ffc on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 66d48ffc on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:f871d6fb-e79f-477b-b5ca-99b939a4d9f7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:f871d6fb-e79f-477b-b5ca-99b939a4d9f7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 69018c26 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 69018c26 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 69018c26 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:b86743b4-04cf-4455-aba9-0bc61ad3815e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:b86743b4-04cf-4455-aba9-0bc61ad3815e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:8e87c593-e4ec-4db5-b533-eab312a2c1cb:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:8e87c593-e4ec-4db5-b533-eab312a2c1cb:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 6d468714 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 6d468714 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 6d468714 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:221b1e18-be90-427f-8a54-9291fe6f5959:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:221b1e18-be90-427f-8a54-9291fe6f5959:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 6f693f32 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 6f693f32 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 6f693f32 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:52fbd8f8-8d38-4465-9ecc-c00e381b7563:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:52fbd8f8-8d38-4465-9ecc-c00e381b7563:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 71a1db42 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 71a1db42 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 71a1db42 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:7a8d4a8c-12b5-4a68-b436-535b8d09f517:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:7a8d4a8c-12b5-4a68-b436-535b8d09f517:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:3de81b6a-badb-4222-8aca-128ebdfec400:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:3de81b6a-badb-4222-8aca-128ebdfec400:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 75dd4066 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 75dd4066 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 75dd4066 on lago-he-basic-iscsi-suite-master-host-0 output: !! Cluster is in GLOBAL MAINTENANCE mode !! --== Host lago-he-basic-iscsi-suite-master-host-0.lago.local (id: 1) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : lago-he-basic-iscsi-suite-master-host-0.lago.local Host ID : 1 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"} Score : 0 stopped : False Local maintenance : False crc32 : 56f93df8 local_conf_timestamp : 2896 Host timestamp : 2895 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2895 (Sun Aug 26 23:29:50 2018) host-id=1 score=0 vm_conf_refresh_time=2896 (Sun Aug 26 23:29:51 2018) conf_on_shared_storage=True maintenance=False state=ReinitializeFSM stopped=False --== Host lago-he-basic-iscsi-suite-master-host-1 (id: 2) status ==-- conf_on_shared_storage : True Status up-to-date : False Hostname : lago-he-basic-iscsi-suite-master-host-1 Host ID : 2 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : ffa35d6d local_conf_timestamp : 2892 Host timestamp : 2891 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2891 (Sun Aug 26 23:29:45 2018) host-id=2 score=3400 vm_conf_refresh_time=2892 (Sun Aug 26 23:29:46 2018) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False !! Cluster is in GLOBAL MAINTENANCE mode !! root: INFO: * Agent is ready. root: INFO: * Starting VM... lago.ssh: DEBUG: start task:0174ef46-3f09-4c57-b0cc-135fc059665a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:0174ef46-3f09-4c57-b0cc-135fc059665a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 764e856e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-start lago.ssh: DEBUG: Command 764e856e on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 764e856e on lago-he-basic-iscsi-suite-master-host-0 output: VM exists and is down, cleaning up and restarting root: INFO: * Command succeeded root: INFO: * Waiting for VM to be UP... lago.ssh: DEBUG: start task:30ee44ce-7f0b-4b16-ba1b-7b6964891efd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:30ee44ce-7f0b-4b16-ba1b-7b6964891efd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 77b7ccda on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 77b7ccda on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 77b7ccda on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2895 (Sun Aug 26 23:29:50 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=2896 (Sun Aug 26 23:29:51 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=ReinitializeFSM\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "56f93df8", "local_conf_timestamp": 2896, "host-ts": 2895}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2891 (Sun Aug 26 23:29:45 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2892 (Sun Aug 26 23:29:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ffa35d6d", "local_conf_timestamp": 2892, "host-ts": 2891}, "global_maintenance": true} lago.ssh: DEBUG: start task:ef26fadc-fd32-4afb-9a06-07e13a3e556f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:ef26fadc-fd32-4afb-9a06-07e13a3e556f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2905 (Sun Aug 26 23:30:00 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2906 (Sun Aug 26 23:30:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9fce1b92", "local_conf_timestamp": 2906, "host-ts": 2905}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2901 (Sun Aug 26 23:29:55 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2902 (Sun Aug 26 23:29:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "68110bea", "local_conf_timestamp": 2902, "host-ts": 2901}, "global_maintenance": true} root: INFO: * VM is UP. root: INFO: * Waiting for engine to start... lago.ssh: DEBUG: start task:22f9ce85-5236-43de-956e-60eedabdee44:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:22f9ce85-5236-43de-956e-60eedabdee44:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2905 (Sun Aug 26 23:30:00 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2906 (Sun Aug 26 23:30:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9fce1b92", "local_conf_timestamp": 2906, "host-ts": 2905}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2901 (Sun Aug 26 23:29:55 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2902 (Sun Aug 26 23:29:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "68110bea", "local_conf_timestamp": 2902, "host-ts": 2901}, "global_maintenance": true} lago.ssh: DEBUG: start task:10d11112-af95-4062-82b5-90a12379d806:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:10d11112-af95-4062-82b5-90a12379d806:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 85234b10 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 85234b10 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 85234b10 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2915 (Sun Aug 26 23:30:10 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2916 (Sun Aug 26 23:30:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69a0d34f", "local_conf_timestamp": 2916, "host-ts": 2915}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2911 (Sun Aug 26 23:30:05 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2913 (Sun Aug 26 23:30:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ff5ba0ab", "local_conf_timestamp": 2913, "host-ts": 2911}, "global_maintenance": true} lago.ssh: DEBUG: start task:8600ff29-4795-4922-9bb5-1ea8cb47aff6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:8600ff29-4795-4922-9bb5-1ea8cb47aff6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 8778cab6 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 8778cab6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 8778cab6 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2915 (Sun Aug 26 23:30:10 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2916 (Sun Aug 26 23:30:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69a0d34f", "local_conf_timestamp": 2916, "host-ts": 2915}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true} lago.ssh: DEBUG: start task:6251e2ec-36ea-42cd-a7cf-018d3efaf73a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:6251e2ec-36ea-42cd-a7cf-018d3efaf73a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 89e10548 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 89e10548 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 89e10548 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2925 (Sun Aug 26 23:30:20 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2927 (Sun Aug 26 23:30:22 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "df65bcff", "local_conf_timestamp": 2927, "host-ts": 2925}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true} lago.ssh: DEBUG: start task:5671a60c-09fc-4f60-97a2-270f9f5b4ca1:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:5671a60c-09fc-4f60-97a2-270f9f5b4ca1:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2925 (Sun Aug 26 23:30:20 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2927 (Sun Aug 26 23:30:22 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "df65bcff", "local_conf_timestamp": 2927, "host-ts": 2925}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true} lago.ssh: DEBUG: start task:ca12f761-8a01-4613-8217-3848165adb82:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:ca12f761-8a01-4613-8217-3848165adb82:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 8efac10e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 8efac10e on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 8efac10e on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2935 (Sun Aug 26 23:30:30 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2937 (Sun Aug 26 23:30:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "41a00e65", "local_conf_timestamp": 2937, "host-ts": 2935}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2932 (Sun Aug 26 23:30:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2934 (Sun Aug 26 23:30:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "903a0802", "local_conf_timestamp": 2934, "host-ts": 2932}, "global_maintenance": true} lago.ssh: DEBUG: start task:2ce2a039-af5f-47e2-b0c8-650b8ca40276:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:2ce2a039-af5f-47e2-b0c8-650b8ca40276:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2935 (Sun Aug 26 23:30:30 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2937 (Sun Aug 26 23:30:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "41a00e65", "local_conf_timestamp": 2937, "host-ts": 2935}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2932 (Sun Aug 26 23:30:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2934 (Sun Aug 26 23:30:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "903a0802", "local_conf_timestamp": 2934, "host-ts": 2932}, "global_maintenance": true} lago.ssh: DEBUG: start task:50565f8c-6b24-4ae7-9eba-bbf9b31b3867:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:50565f8c-6b24-4ae7-9eba-bbf9b31b3867:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 93ca2012 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 93ca2012 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 93ca2012 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true} lago.ssh: DEBUG: start task:8de05a79-ab0e-4659-bc5c-03ab090af777:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:8de05a79-ab0e-4659-bc5c-03ab090af777:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 964c95f4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 964c95f4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 964c95f4 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true} lago.ssh: DEBUG: start task:c85287ef-a0b5-4cea-ab1a-cfc1c4cc97d4:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:c85287ef-a0b5-4cea-ab1a-cfc1c4cc97d4:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 98aaf00c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 98aaf00c on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 98aaf00c on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true} lago.ssh: DEBUG: start task:c1aa3391-00ad-4c4a-a698-304aa8d8e598:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:c1aa3391-00ad-4c4a-a698-304aa8d8e598:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 9b10536e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9b10536e on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 9b10536e on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2955 (Sun Aug 26 23:30:50 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2957 (Sun Aug 26 23:30:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6d9e5c54", "local_conf_timestamp": 2957, "host-ts": 2955}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2952 (Sun Aug 26 23:30:46 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2953 (Sun Aug 26 23:30:47 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2260cf90", "local_conf_timestamp": 2953, "host-ts": 2952}, "global_maintenance": true} lago.ssh: DEBUG: start task:a7f2e7dd-ffdc-4265-a6b0-58c16e4d076f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:a7f2e7dd-ffdc-4265-a6b0-58c16e4d076f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 9d796f1e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9d796f1e on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 9d796f1e on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2955 (Sun Aug 26 23:30:50 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2957 (Sun Aug 26 23:30:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6d9e5c54", "local_conf_timestamp": 2957, "host-ts": 2955}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2952 (Sun Aug 26 23:30:46 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2953 (Sun Aug 26 23:30:47 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2260cf90", "local_conf_timestamp": 2953, "host-ts": 2952}, "global_maintenance": true} lago.ssh: DEBUG: start task:197b6456-4573-4fe5-a902-4d4135afd1c5:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:197b6456-4573-4fe5-a902-4d4135afd1c5:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 9fef209a on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9fef209a on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 9fef209a on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true} lago.ssh: DEBUG: start task:e449730a-24b3-4ba7-a32a-9841d681e8d7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:e449730a-24b3-4ba7-a32a-9841d681e8d7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running a251ea20 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a251ea20 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command a251ea20 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true} lago.ssh: DEBUG: start task:8fa960e8-d3d3-4de3-8673-7ccb2e8e3232:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:8fa960e8-d3d3-4de3-8673-7ccb2e8e3232:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running a4b99e3e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a4b99e3e on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command a4b99e3e on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true} lago.ssh: DEBUG: start task:df444f3d-0958-4232-bbe2-79c9b53d6df7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:df444f3d-0958-4232-bbe2-79c9b53d6df7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running a7386262 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a7386262 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command a7386262 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2975 (Sun Aug 26 23:31:10 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2976 (Sun Aug 26 23:31:11 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1cb13a70", "local_conf_timestamp": 2976, "host-ts": 2975}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2972 (Sun Aug 26 23:31:06 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2973 (Sun Aug 26 23:31:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "398afe7f", "local_conf_timestamp": 2973, "host-ts": 2972}, "global_maintenance": true} lago.ssh: DEBUG: start task:ccb637f6-e93c-474f-8822-99fa3e47e492:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:ccb637f6-e93c-474f-8822-99fa3e47e492:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2975 (Sun Aug 26 23:31:10 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2976 (Sun Aug 26 23:31:11 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1cb13a70", "local_conf_timestamp": 2976, "host-ts": 2975}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2972 (Sun Aug 26 23:31:06 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2973 (Sun Aug 26 23:31:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "398afe7f", "local_conf_timestamp": 2973, "host-ts": 2972}, "global_maintenance": true} lago.ssh: DEBUG: start task:3c133bbf-6ccf-4f97-a708-4902c159a68f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:3c133bbf-6ccf-4f97-a708-4902c159a68f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running ac14d18a on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ac14d18a on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command ac14d18a on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true} lago.ssh: DEBUG: start task:ff5cd16b-0c57-4df9-bbcc-9fd94b5fccb8:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:ff5cd16b-0c57-4df9-bbcc-9fd94b5fccb8:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true} lago.ssh: DEBUG: start task:7fcd4b38-7d15-4000-ad18-d363a3d3a7ab:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:7fcd4b38-7d15-4000-ad18-d363a3d3a7ab:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running b0c63368 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b0c63368 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command b0c63368 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true} lago.ssh: DEBUG: start task:3c20a1c0-ec61-41b6-b247-25ac0093a494:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:3c20a1c0-ec61-41b6-b247-25ac0093a494:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running b30f7738 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b30f7738 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command b30f7738 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7a1c6bf2", "local_conf_timestamp": 2993, "host-ts": 2992}, "global_maintenance": true} lago.ssh: DEBUG: start task:90657a8e-cbc1-4061-823a-c2e3d71635c3:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:90657a8e-cbc1-4061-823a-c2e3d71635c3:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running b555e324 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b555e324 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command b555e324 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f5d8fad1410> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) RuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 53, in restart_he_vm _wait_for_engine_health(host) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 185, in _wait_for_engine_health testlib.assert_true_within_long(lambda: any( File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) 'could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1",\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:a3c71d1c-cb12-4fef-9e6a-d5e0c1b07550:Get ssh client for lago-he-basic-iscsi-suite-master-host-1:\nlago.ssh: DEBUG: end task:a3c71d1c-cb12-4fef-9e6a-d5e0c1b07550:Get ssh client for lago-he-basic-iscsi-suite-master-host-1:\nlago.ssh: DEBUG: Running 43cfc44a on lago-he-basic-iscsi-suite-master-host-1: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 43cfc44a on lago-he-basic-iscsi-suite-master-host-1 returned with 0\nlago.ssh: DEBUG: Command 43cfc44a on lago-he-basic-iscsi-suite-master-host-1 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2805 (Sun Aug 26 23:28:20 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2806 (Sun Aug 26 23:28:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd22cf78", "local_conf_timestamp": 2806, "host-ts": 2805}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2810 (Sun Aug 26 23:28:24 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2812 (Sun Aug 26 23:28:26 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7f7e634d", "local_conf_timestamp": 2812, "host-ts": 2810}, "global_maintenance": true}\n\nroot: INFO: * Shutting down HE VM on host: lago-he-basic-iscsi-suite-master-host-0\nlago.ssh: DEBUG: start task:42afe970-0be5-40d6-834b-245d877c0d1a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:42afe970-0be5-40d6-834b-245d877c0d1a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 445276b0 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-shutdown\nlago.ssh: DEBUG: Command 445276b0 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be down...\nlago.ssh: DEBUG: start task:d92ec1bd-a571-4e86-b3a6-7bedd805567d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:d92ec1bd-a571-4e86-b3a6-7bedd805567d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 44f97960 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 44f97960 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 44f97960 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2805 (Sun Aug 26 23:28:20 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2806 (Sun Aug 26 23:28:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd22cf78", "local_conf_timestamp": 2806, "host-ts": 2805}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2800 (Sun Aug 26 23:28:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2801 (Sun Aug 26 23:28:15 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "063aa920", "local_conf_timestamp": 2801, "host-ts": 2800}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:a1e22b81-c1cf-4b25-a1f7-342286fbe81b:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:a1e22b81-c1cf-4b25-a1f7-342286fbe81b:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:f5948091-c497-4f3b-9761-04d42a2cc421:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:f5948091-c497-4f3b-9761-04d42a2cc421:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 4df162c6 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 4df162c6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 4df162c6 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8b462547-686e-4dbe-8bfb-91d4ba2931dd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:8b462547-686e-4dbe-8bfb-91d4ba2931dd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 504ec626 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 504ec626 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 504ec626 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:f2eef89a-f42e-4c79-9337-72571575f679:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:f2eef89a-f42e-4c79-9337-72571575f679:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 52a4916c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 52a4916c on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 52a4916c on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2835 (Sun Aug 26 23:28:50 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2837 (Sun Aug 26 23:28:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7a4710c6", "local_conf_timestamp": 2837, "host-ts": 2835}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2831 (Sun Aug 26 23:28:45 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2833 (Sun Aug 26 23:28:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fce69ba8", "local_conf_timestamp": 2833, "host-ts": 2831}, "global_maintenance": true}\n\nroot: INFO: * VM is down.\nroot: INFO: * Stopping services...\nlago.ssh: DEBUG: start task:2c25f4a6-f2ef-4cfa-822f-b929c4dfd509:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:2c25f4a6-f2ef-4cfa-822f-b929c4dfd509:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 5322b5b0 on lago-he-basic-iscsi-suite-master-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command 5322b5b0 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nroot: INFO: * Starting services...\nlago.ssh: DEBUG: start task:ee6b4609-b5ac-4f3c-a182-7ed44b8deeb6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ee6b4609-b5ac-4f3c-a182-7ed44b8deeb6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 5a5b1dd6 on lago-he-basic-iscsi-suite-master-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command 5a5b1dd6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nroot: INFO: * Waiting for agent to be ready...\nlago.ssh: DEBUG: start task:facd8e55-61ed-4fb7-b9fe-c3ad705d968a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:facd8e55-61ed-4fb7-b9fe-c3ad705d968a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:3eb14b7c-074b-4c11-a117-392f8b162c2e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:3eb14b7c-074b-4c11-a117-392f8b162c2e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 62a124ea on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 62a124ea on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 62a124ea on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:c78b3a53-f7bb-480e-839e-b57e9ca7eb4d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:c78b3a53-f7bb-480e-839e-b57e9ca7eb4d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:ca525d74-b2b6-4297-ad54-b044bc452568:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ca525d74-b2b6-4297-ad54-b044bc452568:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 66d48ffc on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 66d48ffc on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 66d48ffc on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:f871d6fb-e79f-477b-b5ca-99b939a4d9f7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:f871d6fb-e79f-477b-b5ca-99b939a4d9f7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 69018c26 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 69018c26 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 69018c26 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:b86743b4-04cf-4455-aba9-0bc61ad3815e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:b86743b4-04cf-4455-aba9-0bc61ad3815e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:8e87c593-e4ec-4db5-b533-eab312a2c1cb:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:8e87c593-e4ec-4db5-b533-eab312a2c1cb:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 6d468714 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 6d468714 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 6d468714 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:221b1e18-be90-427f-8a54-9291fe6f5959:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:221b1e18-be90-427f-8a54-9291fe6f5959:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 6f693f32 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 6f693f32 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 6f693f32 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:52fbd8f8-8d38-4465-9ecc-c00e381b7563:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:52fbd8f8-8d38-4465-9ecc-c00e381b7563:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 71a1db42 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 71a1db42 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 71a1db42 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:7a8d4a8c-12b5-4a68-b436-535b8d09f517:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:7a8d4a8c-12b5-4a68-b436-535b8d09f517:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:3de81b6a-badb-4222-8aca-128ebdfec400:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:3de81b6a-badb-4222-8aca-128ebdfec400:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 75dd4066 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 75dd4066 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 75dd4066 on lago-he-basic-iscsi-suite-master-host-0 output:\n \n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\n\n--== Host lago-he-basic-iscsi-suite-master-host-0.lago.local (id: 1) status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : True\nHostname : lago-he-basic-iscsi-suite-master-host-0.lago.local\nHost ID : 1\nEngine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}\nScore : 0\nstopped : False\nLocal maintenance : False\ncrc32 : 56f93df8\nlocal_conf_timestamp : 2896\nHost timestamp : 2895\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2895 (Sun Aug 26 23:29:50 2018)\n\thost-id=1\n\tscore=0\n\tvm_conf_refresh_time=2896 (Sun Aug 26 23:29:51 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=ReinitializeFSM\n\tstopped=False\n\n\n--== Host lago-he-basic-iscsi-suite-master-host-1 (id: 2) status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : False\nHostname : lago-he-basic-iscsi-suite-master-host-1\nHost ID : 2\nEngine status : unknown stale-data\nScore : 3400\nstopped : False\nLocal maintenance : False\ncrc32 : ffa35d6d\nlocal_conf_timestamp : 2892\nHost timestamp : 2891\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2891 (Sun Aug 26 23:29:45 2018)\n\thost-id=2\n\tscore=3400\n\tvm_conf_refresh_time=2892 (Sun Aug 26 23:29:46 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=GlobalMaintenance\n\tstopped=False\n\n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\nroot: INFO: * Agent is ready.\nroot: INFO: * Starting VM...\nlago.ssh: DEBUG: start task:0174ef46-3f09-4c57-b0cc-135fc059665a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:0174ef46-3f09-4c57-b0cc-135fc059665a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 764e856e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-start\nlago.ssh: DEBUG: Command 764e856e on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 764e856e on lago-he-basic-iscsi-suite-master-host-0 output:\n VM exists and is down, cleaning up and restarting\n\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be UP...\nlago.ssh: DEBUG: start task:30ee44ce-7f0b-4b16-ba1b-7b6964891efd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:30ee44ce-7f0b-4b16-ba1b-7b6964891efd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 77b7ccda on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 77b7ccda on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 77b7ccda on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2895 (Sun Aug 26 23:29:50 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=2896 (Sun Aug 26 23:29:51 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=ReinitializeFSM\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "56f93df8", "local_conf_timestamp": 2896, "host-ts": 2895}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2891 (Sun Aug 26 23:29:45 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2892 (Sun Aug 26 23:29:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ffa35d6d", "local_conf_timestamp": 2892, "host-ts": 2891}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ef26fadc-fd32-4afb-9a06-07e13a3e556f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ef26fadc-fd32-4afb-9a06-07e13a3e556f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2905 (Sun Aug 26 23:30:00 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2906 (Sun Aug 26 23:30:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9fce1b92", "local_conf_timestamp": 2906, "host-ts": 2905}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2901 (Sun Aug 26 23:29:55 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2902 (Sun Aug 26 23:29:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "68110bea", "local_conf_timestamp": 2902, "host-ts": 2901}, "global_maintenance": true}\n\nroot: INFO: * VM is UP.\nroot: INFO: * Waiting for engine to start...\nlago.ssh: DEBUG: start task:22f9ce85-5236-43de-956e-60eedabdee44:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:22f9ce85-5236-43de-956e-60eedabdee44:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2905 (Sun Aug 26 23:30:00 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2906 (Sun Aug 26 23:30:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9fce1b92", "local_conf_timestamp": 2906, "host-ts": 2905}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2901 (Sun Aug 26 23:29:55 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2902 (Sun Aug 26 23:29:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "68110bea", "local_conf_timestamp": 2902, "host-ts": 2901}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:10d11112-af95-4062-82b5-90a12379d806:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:10d11112-af95-4062-82b5-90a12379d806:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 85234b10 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 85234b10 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 85234b10 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2915 (Sun Aug 26 23:30:10 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2916 (Sun Aug 26 23:30:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69a0d34f", "local_conf_timestamp": 2916, "host-ts": 2915}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2911 (Sun Aug 26 23:30:05 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2913 (Sun Aug 26 23:30:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ff5ba0ab", "local_conf_timestamp": 2913, "host-ts": 2911}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8600ff29-4795-4922-9bb5-1ea8cb47aff6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:8600ff29-4795-4922-9bb5-1ea8cb47aff6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 8778cab6 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 8778cab6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 8778cab6 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2915 (Sun Aug 26 23:30:10 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2916 (Sun Aug 26 23:30:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69a0d34f", "local_conf_timestamp": 2916, "host-ts": 2915}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:6251e2ec-36ea-42cd-a7cf-018d3efaf73a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:6251e2ec-36ea-42cd-a7cf-018d3efaf73a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 89e10548 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 89e10548 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 89e10548 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2925 (Sun Aug 26 23:30:20 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2927 (Sun Aug 26 23:30:22 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "df65bcff", "local_conf_timestamp": 2927, "host-ts": 2925}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:5671a60c-09fc-4f60-97a2-270f9f5b4ca1:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:5671a60c-09fc-4f60-97a2-270f9f5b4ca1:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2925 (Sun Aug 26 23:30:20 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2927 (Sun Aug 26 23:30:22 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "df65bcff", "local_conf_timestamp": 2927, "host-ts": 2925}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ca12f761-8a01-4613-8217-3848165adb82:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ca12f761-8a01-4613-8217-3848165adb82:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 8efac10e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 8efac10e on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 8efac10e on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2935 (Sun Aug 26 23:30:30 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2937 (Sun Aug 26 23:30:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "41a00e65", "local_conf_timestamp": 2937, "host-ts": 2935}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2932 (Sun Aug 26 23:30:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2934 (Sun Aug 26 23:30:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "903a0802", "local_conf_timestamp": 2934, "host-ts": 2932}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:2ce2a039-af5f-47e2-b0c8-650b8ca40276:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:2ce2a039-af5f-47e2-b0c8-650b8ca40276:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2935 (Sun Aug 26 23:30:30 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2937 (Sun Aug 26 23:30:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "41a00e65", "local_conf_timestamp": 2937, "host-ts": 2935}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2932 (Sun Aug 26 23:30:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2934 (Sun Aug 26 23:30:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "903a0802", "local_conf_timestamp": 2934, "host-ts": 2932}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:50565f8c-6b24-4ae7-9eba-bbf9b31b3867:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:50565f8c-6b24-4ae7-9eba-bbf9b31b3867:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 93ca2012 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 93ca2012 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 93ca2012 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8de05a79-ab0e-4659-bc5c-03ab090af777:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:8de05a79-ab0e-4659-bc5c-03ab090af777:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 964c95f4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 964c95f4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 964c95f4 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:c85287ef-a0b5-4cea-ab1a-cfc1c4cc97d4:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:c85287ef-a0b5-4cea-ab1a-cfc1c4cc97d4:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 98aaf00c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 98aaf00c on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 98aaf00c on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:c1aa3391-00ad-4c4a-a698-304aa8d8e598:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:c1aa3391-00ad-4c4a-a698-304aa8d8e598:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 9b10536e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9b10536e on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 9b10536e on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2955 (Sun Aug 26 23:30:50 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2957 (Sun Aug 26 23:30:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6d9e5c54", "local_conf_timestamp": 2957, "host-ts": 2955}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2952 (Sun Aug 26 23:30:46 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2953 (Sun Aug 26 23:30:47 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2260cf90", "local_conf_timestamp": 2953, "host-ts": 2952}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:a7f2e7dd-ffdc-4265-a6b0-58c16e4d076f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:a7f2e7dd-ffdc-4265-a6b0-58c16e4d076f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 9d796f1e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9d796f1e on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 9d796f1e on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2955 (Sun Aug 26 23:30:50 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2957 (Sun Aug 26 23:30:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6d9e5c54", "local_conf_timestamp": 2957, "host-ts": 2955}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2952 (Sun Aug 26 23:30:46 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2953 (Sun Aug 26 23:30:47 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2260cf90", "local_conf_timestamp": 2953, "host-ts": 2952}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:197b6456-4573-4fe5-a902-4d4135afd1c5:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:197b6456-4573-4fe5-a902-4d4135afd1c5:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 9fef209a on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9fef209a on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 9fef209a on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:e449730a-24b3-4ba7-a32a-9841d681e8d7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:e449730a-24b3-4ba7-a32a-9841d681e8d7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running a251ea20 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a251ea20 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a251ea20 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8fa960e8-d3d3-4de3-8673-7ccb2e8e3232:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:8fa960e8-d3d3-4de3-8673-7ccb2e8e3232:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running a4b99e3e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a4b99e3e on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a4b99e3e on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:df444f3d-0958-4232-bbe2-79c9b53d6df7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:df444f3d-0958-4232-bbe2-79c9b53d6df7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running a7386262 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a7386262 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a7386262 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2975 (Sun Aug 26 23:31:10 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2976 (Sun Aug 26 23:31:11 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1cb13a70", "local_conf_timestamp": 2976, "host-ts": 2975}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2972 (Sun Aug 26 23:31:06 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2973 (Sun Aug 26 23:31:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "398afe7f", "local_conf_timestamp": 2973, "host-ts": 2972}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ccb637f6-e93c-474f-8822-99fa3e47e492:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ccb637f6-e93c-474f-8822-99fa3e47e492:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2975 (Sun Aug 26 23:31:10 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2976 (Sun Aug 26 23:31:11 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1cb13a70", "local_conf_timestamp": 2976, "host-ts": 2975}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2972 (Sun Aug 26 23:31:06 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2973 (Sun Aug 26 23:31:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "398afe7f", "local_conf_timestamp": 2973, "host-ts": 2972}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:3c133bbf-6ccf-4f97-a708-4902c159a68f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:3c133bbf-6ccf-4f97-a708-4902c159a68f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running ac14d18a on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ac14d18a on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ac14d18a on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ff5cd16b-0c57-4df9-bbcc-9fd94b5fccb8:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ff5cd16b-0c57-4df9-bbcc-9fd94b5fccb8:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:7fcd4b38-7d15-4000-ad18-d363a3d3a7ab:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:7fcd4b38-7d15-4000-ad18-d363a3d3a7ab:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running b0c63368 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b0c63368 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command b0c63368 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:3c20a1c0-ec61-41b6-b247-25ac0093a494:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:3c20a1c0-ec61-41b6-b247-25ac0093a494:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running b30f7738 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b30f7738 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command b30f7738 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7a1c6bf2", "local_conf_timestamp": 2993, "host-ts": 2992}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:90657a8e-cbc1-4061-823a-c2e3d71635c3:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:90657a8e-cbc1-4061-823a-c2e3d71635c3:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running b555e324 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b555e324 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command b555e324 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1",\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f5d8fad1410>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda>\n for k, v in _get_he_status(host).items()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status\n raise RuntimeError(\'could not parse JSON: %s\' % ret.out)\nRuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1",\n--------------------- >> end captured logging << ---------------------'

6 years, 2 months

1
2
0 / 0

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #1884

by jenkins＠jenkins.phx.ovirt.org

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 3afc9ba57755a1b2574f7a177184eb66d5f6a90d (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 3afc9ba57755a1b2574f7a177184eb66d5f6a90d Commit message: "ovirt-release: drop non working jobs" > git rev-list --no-walk 3afc9ba57755a1b2574f7a177184eb66d5f6a90d # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins1863032023757664663.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7' Build step 'Execute shell' marked build as failure

6 years, 2 months

1
1
0 / 0

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build # 639 - Failure!

by jenkins＠jenkins.phx.ovirt.org

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/639/ Build Number: 639 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #639 [Roy Golan] Introduce openshift-on-ovirt ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.snapshot_merge Error Message: status: 409 reason: Conflict detail: Cannot create Snapshot. Snapshot is currently being created for VM vm0. Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 69, in wrapper return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/004_basic_sanity.py", line 194, in snapshot_merge api.vms.get(VM0_NAME).snapshots.add(dead_snap2_params) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 34511, in add headers={"Correlation-Id":correlation_id, "Expect":expect} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add return self.request('POST', url, body, headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) RequestError: status: 409 reason: Conflict detail: Cannot create Snapshot. Snapshot is currently being created for VM vm0.

6 years, 2 months

1
1
0 / 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Infra August 2018