August 2018 - Infra - oVirt List Archives

[JIRA] (OVIRT-2454) fcraw mirror sync job failing for several days
by Eyal Edri (oVirt JIRA) 28 Aug '18

28 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2454?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2454: ---------------------------------- Do we still need it? I don't recall any project still uses it. > fcraw mirror sync job failing for several days > ---------------------------------------------- > > Key: OVIRT-2454 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2454 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Evgheni Dereveanchin > Assignee: infra > > The FC Rawhide sync joob is failing for more than a week: > https://jenkins.ovirt.org/job/system-sync_mirrors-fedora-base-fcraw-x86_64/ > This brings unnecessary load to the mirror server so opening a ticket to investigate the reason -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2454) fcraw mirror sync job failing for several days
by Evgheni Dereveanchin (oVirt JIRA) 28 Aug '18

28 Aug '18

Evgheni Dereveanchin created OVIRT-2454: ------------------------------------------- Summary: fcraw mirror sync job failing for several days Key: OVIRT-2454 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2454 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Evgheni Dereveanchin Assignee: infra The FC Rawhide sync joob is failing for more than a week: https://jenkins.ovirt.org/job/system-sync_mirrors-fedora-base-fcraw-x86_64/ This brings unnecessary load to the mirror server so opening a ticket to investigate the reason -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2452) Re: New IP addresses for mirror ftp.plusline.net
by Evgheni Dereveanchin (oVirt JIRA) 28 Aug '18

28 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2452?page=com.atlassian.jira.… ] Evgheni Dereveanchin commented on OVIRT-2452: --------------------------------------------- Request logged to white-list the new IP. Will update you once that is done. > Re: New IP addresses for mirror ftp.plusline.net > ------------------------------------------------ > > Key: OVIRT-2452 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2452 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Eyal Edri > Assignee: Evgheni Dereveanchin > > Evgheni, can you help? > Also forwarding to infra-support to open a JIRA ticket. > On Sat, Aug 25, 2018 at 7:07 PM Jan Siml <jsiml(a)plusline.net> wrote: > > Hello, > > > > we had to rebuild our mirror server and it is now available at new IP > > addresses. Since these are also used for the synchronization of > > resources.ovirt.org, I would like to ask you to unlock the new IP > > addresses. They are as follows: > > > > 82.98.66.54 > > 2a02:2e0:414:102:d::54 > > > > The SSH key used remains the same. > > > > Kind regards > > > > Jan > > _______________________________________________ > > Infra mailing list -- infra(a)ovirt.org > > To unsubscribe send an email to infra-leave(a)ovirt.org > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > oVirt Code of Conduct: > > https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: > > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/F4VSLEZLX5KMR… > > > -- > Eyal edri > MANAGER > RHV DevOps > EMEA VIRTUALIZATION R&D > Red Hat EMEA <https://www.redhat.com/> > <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2452) Re: New IP addresses for mirror ftp.plusline.net
by Evgheni Dereveanchin (oVirt JIRA) 28 Aug '18

28 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2452?page=com.atlassian.jira.… ] Evgheni Dereveanchin reassigned OVIRT-2452: ------------------------------------------- Assignee: Evgheni Dereveanchin (was: infra) > Re: New IP addresses for mirror ftp.plusline.net > ------------------------------------------------ > > Key: OVIRT-2452 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2452 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Eyal Edri > Assignee: Evgheni Dereveanchin > > Evgheni, can you help? > Also forwarding to infra-support to open a JIRA ticket. > On Sat, Aug 25, 2018 at 7:07 PM Jan Siml <jsiml(a)plusline.net> wrote: > > Hello, > > > > we had to rebuild our mirror server and it is now available at new IP > > addresses. Since these are also used for the synchronization of > > resources.ovirt.org, I would like to ask you to unlock the new IP > > addresses. They are as follows: > > > > 82.98.66.54 > > 2a02:2e0:414:102:d::54 > > > > The SSH key used remains the same. > > > > Kind regards > > > > Jan > > _______________________________________________ > > Infra mailing list -- infra(a)ovirt.org > > To unsubscribe send an email to infra-leave(a)ovirt.org > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > oVirt Code of Conduct: > > https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: > > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/F4VSLEZLX5KMR… > > > -- > Eyal edri > MANAGER > RHV DevOps > EMEA VIRTUALIZATION R&D > Red Hat EMEA <https://www.redhat.com/> > <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2453) vdsm_{master,4.2} check-merge to run on bare metal slaves on CI
by Eyal Edri (oVirt JIRA) 28 Aug '18

28 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2453?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2453: ---------------------------------- [~dbelenky(a)redhat.com] [~gbenhaim(a)redhat.com] is it possible to try running it on the new containers? > vdsm_{master,4.2} check-merge to run on bare metal slaves on CI > --------------------------------------------------------------- > > Key: OVIRT-2453 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2453 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Edward Haas > Assignee: infra > > Hello, > VDSM check-merge is using lago VM/s to run functional tests, mainly in the > network domain. > Unfortunately, when running on a VM slave, the tests are running in a > nested VM which seems to be unstable (the nested VM is getting > stuck/crashes). > Here are some recent failures: > http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/247/ > http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/249/ > Could you please run this job on a bare metal slave? > Thanks, > Edy. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[oVirt Jenkins] ovirt-node-ng-image_master_build-artifacts-fc28-x86_64 - Build # 34 - Failure!
by jenkins＠jenkins.phx.ovirt.org 28 Aug '18

28 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-fc2… Build: http://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-fc2… Build Number: 34 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #34 [Yuval Turgeman] stdci: move build-artifacts to a v1 job ----------------- Failed Tests: ----------------- No tests ran.

1 2

[JIRA] (OVIRT-2453) vdsm_{master,4.2} check-merge to run on bare metal slaves on CI
by Daniel Belenky (oVirt JIRA) 28 Aug '18

28 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2453?page=com.atlassian.jira.… ] Daniel Belenky commented on OVIRT-2453: --------------------------------------- On Tue, Aug 28, 2018 at 8:23 AM Edward Haas <ehaas(a)redhat.com> wrote: > Hello, > > VDSM check-merge is using lago VM/s to run functional tests, mainly in the > network domain. > Unfortunately, when running on a VM slave, the tests are running in a > nested VM which seems to be unstable (the nested VM is getting > stuck/crashes). > > Here are some recent failures: > http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/247/ > http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/249/ > > Could you please run this job on a bare metal slave? > > Thanks, > Edy. > -- DANIEL BELENKY > vdsm_{master,4.2} check-merge to run on bare metal slaves on CI > --------------------------------------------------------------- > > Key: OVIRT-2453 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2453 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Edward Haas > Assignee: infra > > Hello, > VDSM check-merge is using lago VM/s to run functional tests, mainly in the > network domain. > Unfortunately, when running on a VM slave, the tests are running in a > nested VM which seems to be unstable (the nested VM is getting > stuck/crashes). > Here are some recent failures: > http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/247/ > http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/249/ > Could you please run this job on a bare metal slave? > Thanks, > Edy. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #1886
by jenkins＠jenkins.phx.ovirt.org 28 Aug '18

28 Aug '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 8aa2a8538d0bc192415e0eac6c831cbe94d68793 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 8aa2a8538d0bc192415e0eac6c831cbe94d68793 Commit message: "mock_runner: Set no_proxy for proxied configs" > git rev-list --no-walk 8aa2a8538d0bc192415e0eac6c831cbe94d68793 # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins2807930280698784854.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7' Build step 'Execute shell' marked build as failure

1 1

[JIRA] (OVIRT-2453) vdsm_{master,4.2} check-merge to run on bare metal slaves on CI
by Edward Haas (oVirt JIRA) 28 Aug '18

28 Aug '18

Edward Haas created OVIRT-2453: ---------------------------------- Summary: vdsm_{master,4.2} check-merge to run on bare metal slaves on CI Key: OVIRT-2453 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2453 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Edward Haas Assignee: infra Hello, VDSM check-merge is using lago VM/s to run functional tests, mainly in the network domain. Unfortunately, when running on a VM slave, the tests are running in a nested VM which seems to be unstable (the nested VM is getting stuck/crashes). Here are some recent failures: http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/247/ http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/249/ Could you please run this job on a bare metal slave? Thanks, Edy. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

Build failed in Jenkins: system-sync_mirrors-glusterfs-4.1-test-el7-x86_64 #1
by jenkins＠jenkins.phx.ovirt.org 28 Aug '18

28 Aug '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-glusterfs-4.1-test-el7-x86…> ------------------------------------------ Started by user Ehud Yonasi [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-glusterfs-4.1-test-el7-x86…> Cloning the remote Git repository Cloning repository http://gerrit.ovirt.org/jenkins.git > git init <http://jenkins.ovirt.org/job/system-sync_mirrors-glusterfs-4.1-test-el7-x86…> # timeout=10 Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 No valid HEAD. Skipping the resetting > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 8aa2a8538d0bc192415e0eac6c831cbe94d68793 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 8aa2a8538d0bc192415e0eac6c831cbe94d68793 Commit message: "mock_runner: Set no_proxy for proxied configs" First time build. Skipping changelog. [system-sync_mirrors-glusterfs-4.1-test-el7-x86_64] $ /bin/bash -xe /tmp/jenkins5612468679084601662.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror glusterfs-4.1-test-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Warning: cannot find repository glusterfs-4.1-test-el7 No repositories found Build step 'Execute shell' marked build as failure

2 4

oVirt infra daily report - unstable production jobs - 565
by jenkins＠jenkins.phx.ovirt.org 28 Aug '18

28 Aug '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/565//artifact/exported-a… Cheers, Jenkins

1 0

[CQ]: 93925,2 (vdsm) failed "ovirt-4.2" system tests
by oVirt Jenkins 27 Aug '18

27 Aug '18

Change 93925,2 (vdsm) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/93925/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2993/

1 0

[oVirt Jenkins] ovirt-system-tests_he-basic-iscsi-suite-master - Build # 464 - Failure!
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18

27 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master… Build Number: 464 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #464 [Roy Golan] Introduce openshift-on-ovirt ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 008_restart_he_vm.restart_he_vm Error Message: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", -------------------- >> begin captured logging << -------------------- lago.ssh: DEBUG: start task:a3c71d1c-cb12-4fef-9e6a-d5e0c1b07550:Get ssh client for lago-he-basic-iscsi-suite-master-host-1: lago.ssh: DEBUG: end task:a3c71d1c-cb12-4fef-9e6a-d5e0c1b07550:Get ssh client for lago-he-basic-iscsi-suite-master-host-1: lago.ssh: DEBUG: Running 43cfc44a on lago-he-basic-iscsi-suite-master-host-1: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 43cfc44a on lago-he-basic-iscsi-suite-master-host-1 returned with 0 lago.ssh: DEBUG: Command 43cfc44a on lago-he-basic-iscsi-suite-master-host-1 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2805 (Sun Aug 26 23:28:20 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2806 (Sun Aug 26 23:28:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd22cf78", "local_conf_timestamp": 2806, "host-ts": 2805}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2810 (Sun Aug 26 23:28:24 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2812 (Sun Aug 26 23:28:26 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7f7e634d", "local_conf_timestamp": 2812, "host-ts": 2810}, "global_maintenance": true} root: INFO: * Shutting down HE VM on host: lago-he-basic-iscsi-suite-master-host-0 lago.ssh: DEBUG: start task:42afe970-0be5-40d6-834b-245d877c0d1a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:42afe970-0be5-40d6-834b-245d877c0d1a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 445276b0 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-shutdown lago.ssh: DEBUG: Command 445276b0 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 root: INFO: * Command succeeded root: INFO: * Waiting for VM to be down... lago.ssh: DEBUG: start task:d92ec1bd-a571-4e86-b3a6-7bedd805567d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:d92ec1bd-a571-4e86-b3a6-7bedd805567d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 44f97960 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 44f97960 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 44f97960 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2805 (Sun Aug 26 23:28:20 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2806 (Sun Aug 26 23:28:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd22cf78", "local_conf_timestamp": 2806, "host-ts": 2805}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2800 (Sun Aug 26 23:28:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2801 (Sun Aug 26 23:28:15 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "063aa920", "local_conf_timestamp": 2801, "host-ts": 2800}, "global_maintenance": false} lago.ssh: DEBUG: start task:a1e22b81-c1cf-4b25-a1f7-342286fbe81b:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:a1e22b81-c1cf-4b25-a1f7-342286fbe81b:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true} lago.ssh: DEBUG: start task:f5948091-c497-4f3b-9761-04d42a2cc421:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:f5948091-c497-4f3b-9761-04d42a2cc421:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 4df162c6 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 4df162c6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 4df162c6 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true} lago.ssh: DEBUG: start task:8b462547-686e-4dbe-8bfb-91d4ba2931dd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:8b462547-686e-4dbe-8bfb-91d4ba2931dd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 504ec626 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 504ec626 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 504ec626 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true} lago.ssh: DEBUG: start task:f2eef89a-f42e-4c79-9337-72571575f679:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:f2eef89a-f42e-4c79-9337-72571575f679:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 52a4916c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 52a4916c on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 52a4916c on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2835 (Sun Aug 26 23:28:50 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2837 (Sun Aug 26 23:28:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7a4710c6", "local_conf_timestamp": 2837, "host-ts": 2835}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2831 (Sun Aug 26 23:28:45 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2833 (Sun Aug 26 23:28:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fce69ba8", "local_conf_timestamp": 2833, "host-ts": 2831}, "global_maintenance": true} root: INFO: * VM is down. root: INFO: * Stopping services... lago.ssh: DEBUG: start task:2c25f4a6-f2ef-4cfa-822f-b929c4dfd509:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:2c25f4a6-f2ef-4cfa-822f-b929c4dfd509:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 5322b5b0 on lago-he-basic-iscsi-suite-master-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command 5322b5b0 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 root: INFO: * Starting services... lago.ssh: DEBUG: start task:ee6b4609-b5ac-4f3c-a182-7ed44b8deeb6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:ee6b4609-b5ac-4f3c-a182-7ed44b8deeb6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 5a5b1dd6 on lago-he-basic-iscsi-suite-master-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command 5a5b1dd6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 root: INFO: * Waiting for agent to be ready... lago.ssh: DEBUG: start task:facd8e55-61ed-4fb7-b9fe-c3ad705d968a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:facd8e55-61ed-4fb7-b9fe-c3ad705d968a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:3eb14b7c-074b-4c11-a117-392f8b162c2e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:3eb14b7c-074b-4c11-a117-392f8b162c2e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 62a124ea on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 62a124ea on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 62a124ea on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:c78b3a53-f7bb-480e-839e-b57e9ca7eb4d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:c78b3a53-f7bb-480e-839e-b57e9ca7eb4d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:ca525d74-b2b6-4297-ad54-b044bc452568:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:ca525d74-b2b6-4297-ad54-b044bc452568:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 66d48ffc on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 66d48ffc on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 66d48ffc on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:f871d6fb-e79f-477b-b5ca-99b939a4d9f7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:f871d6fb-e79f-477b-b5ca-99b939a4d9f7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 69018c26 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 69018c26 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 69018c26 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:b86743b4-04cf-4455-aba9-0bc61ad3815e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:b86743b4-04cf-4455-aba9-0bc61ad3815e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:8e87c593-e4ec-4db5-b533-eab312a2c1cb:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:8e87c593-e4ec-4db5-b533-eab312a2c1cb:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 6d468714 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 6d468714 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 6d468714 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:221b1e18-be90-427f-8a54-9291fe6f5959:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:221b1e18-be90-427f-8a54-9291fe6f5959:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 6f693f32 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 6f693f32 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 6f693f32 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:52fbd8f8-8d38-4465-9ecc-c00e381b7563:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:52fbd8f8-8d38-4465-9ecc-c00e381b7563:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 71a1db42 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 71a1db42 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 71a1db42 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:7a8d4a8c-12b5-4a68-b436-535b8d09f517:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:7a8d4a8c-12b5-4a68-b436-535b8d09f517:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:3de81b6a-badb-4222-8aca-128ebdfec400:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:3de81b6a-badb-4222-8aca-128ebdfec400:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 75dd4066 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 75dd4066 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 75dd4066 on lago-he-basic-iscsi-suite-master-host-0 output: !! Cluster is in GLOBAL MAINTENANCE mode !! --== Host lago-he-basic-iscsi-suite-master-host-0.lago.local (id: 1) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : lago-he-basic-iscsi-suite-master-host-0.lago.local Host ID : 1 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"} Score : 0 stopped : False Local maintenance : False crc32 : 56f93df8 local_conf_timestamp : 2896 Host timestamp : 2895 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2895 (Sun Aug 26 23:29:50 2018) host-id=1 score=0 vm_conf_refresh_time=2896 (Sun Aug 26 23:29:51 2018) conf_on_shared_storage=True maintenance=False state=ReinitializeFSM stopped=False --== Host lago-he-basic-iscsi-suite-master-host-1 (id: 2) status ==-- conf_on_shared_storage : True Status up-to-date : False Hostname : lago-he-basic-iscsi-suite-master-host-1 Host ID : 2 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : ffa35d6d local_conf_timestamp : 2892 Host timestamp : 2891 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2891 (Sun Aug 26 23:29:45 2018) host-id=2 score=3400 vm_conf_refresh_time=2892 (Sun Aug 26 23:29:46 2018) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False !! Cluster is in GLOBAL MAINTENANCE mode !! root: INFO: * Agent is ready. root: INFO: * Starting VM... lago.ssh: DEBUG: start task:0174ef46-3f09-4c57-b0cc-135fc059665a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:0174ef46-3f09-4c57-b0cc-135fc059665a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 764e856e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-start lago.ssh: DEBUG: Command 764e856e on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 764e856e on lago-he-basic-iscsi-suite-master-host-0 output: VM exists and is down, cleaning up and restarting root: INFO: * Command succeeded root: INFO: * Waiting for VM to be UP... lago.ssh: DEBUG: start task:30ee44ce-7f0b-4b16-ba1b-7b6964891efd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:30ee44ce-7f0b-4b16-ba1b-7b6964891efd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 77b7ccda on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 77b7ccda on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 77b7ccda on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2895 (Sun Aug 26 23:29:50 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=2896 (Sun Aug 26 23:29:51 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=ReinitializeFSM\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "56f93df8", "local_conf_timestamp": 2896, "host-ts": 2895}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2891 (Sun Aug 26 23:29:45 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2892 (Sun Aug 26 23:29:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ffa35d6d", "local_conf_timestamp": 2892, "host-ts": 2891}, "global_maintenance": true} lago.ssh: DEBUG: start task:ef26fadc-fd32-4afb-9a06-07e13a3e556f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:ef26fadc-fd32-4afb-9a06-07e13a3e556f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2905 (Sun Aug 26 23:30:00 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2906 (Sun Aug 26 23:30:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9fce1b92", "local_conf_timestamp": 2906, "host-ts": 2905}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2901 (Sun Aug 26 23:29:55 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2902 (Sun Aug 26 23:29:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "68110bea", "local_conf_timestamp": 2902, "host-ts": 2901}, "global_maintenance": true} root: INFO: * VM is UP. root: INFO: * Waiting for engine to start... lago.ssh: DEBUG: start task:22f9ce85-5236-43de-956e-60eedabdee44:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:22f9ce85-5236-43de-956e-60eedabdee44:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2905 (Sun Aug 26 23:30:00 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2906 (Sun Aug 26 23:30:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9fce1b92", "local_conf_timestamp": 2906, "host-ts": 2905}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2901 (Sun Aug 26 23:29:55 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2902 (Sun Aug 26 23:29:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "68110bea", "local_conf_timestamp": 2902, "host-ts": 2901}, "global_maintenance": true} lago.ssh: DEBUG: start task:10d11112-af95-4062-82b5-90a12379d806:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:10d11112-af95-4062-82b5-90a12379d806:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 85234b10 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 85234b10 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 85234b10 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2915 (Sun Aug 26 23:30:10 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2916 (Sun Aug 26 23:30:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69a0d34f", "local_conf_timestamp": 2916, "host-ts": 2915}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2911 (Sun Aug 26 23:30:05 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2913 (Sun Aug 26 23:30:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ff5ba0ab", "local_conf_timestamp": 2913, "host-ts": 2911}, "global_maintenance": true} lago.ssh: DEBUG: start task:8600ff29-4795-4922-9bb5-1ea8cb47aff6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:8600ff29-4795-4922-9bb5-1ea8cb47aff6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 8778cab6 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 8778cab6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 8778cab6 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2915 (Sun Aug 26 23:30:10 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2916 (Sun Aug 26 23:30:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69a0d34f", "local_conf_timestamp": 2916, "host-ts": 2915}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true} lago.ssh: DEBUG: start task:6251e2ec-36ea-42cd-a7cf-018d3efaf73a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:6251e2ec-36ea-42cd-a7cf-018d3efaf73a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 89e10548 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 89e10548 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 89e10548 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2925 (Sun Aug 26 23:30:20 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2927 (Sun Aug 26 23:30:22 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "df65bcff", "local_conf_timestamp": 2927, "host-ts": 2925}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true} lago.ssh: DEBUG: start task:5671a60c-09fc-4f60-97a2-270f9f5b4ca1:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:5671a60c-09fc-4f60-97a2-270f9f5b4ca1:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2925 (Sun Aug 26 23:30:20 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2927 (Sun Aug 26 23:30:22 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "df65bcff", "local_conf_timestamp": 2927, "host-ts": 2925}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true} lago.ssh: DEBUG: start task:ca12f761-8a01-4613-8217-3848165adb82:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:ca12f761-8a01-4613-8217-3848165adb82:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 8efac10e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 8efac10e on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 8efac10e on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2935 (Sun Aug 26 23:30:30 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2937 (Sun Aug 26 23:30:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "41a00e65", "local_conf_timestamp": 2937, "host-ts": 2935}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2932 (Sun Aug 26 23:30:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2934 (Sun Aug 26 23:30:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "903a0802", "local_conf_timestamp": 2934, "host-ts": 2932}, "global_maintenance": true} lago.ssh: DEBUG: start task:2ce2a039-af5f-47e2-b0c8-650b8ca40276:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:2ce2a039-af5f-47e2-b0c8-650b8ca40276:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2935 (Sun Aug 26 23:30:30 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2937 (Sun Aug 26 23:30:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "41a00e65", "local_conf_timestamp": 2937, "host-ts": 2935}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2932 (Sun Aug 26 23:30:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2934 (Sun Aug 26 23:30:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "903a0802", "local_conf_timestamp": 2934, "host-ts": 2932}, "global_maintenance": true} lago.ssh: DEBUG: start task:50565f8c-6b24-4ae7-9eba-bbf9b31b3867:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:50565f8c-6b24-4ae7-9eba-bbf9b31b3867:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 93ca2012 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 93ca2012 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 93ca2012 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true} lago.ssh: DEBUG: start task:8de05a79-ab0e-4659-bc5c-03ab090af777:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:8de05a79-ab0e-4659-bc5c-03ab090af777:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 964c95f4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 964c95f4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 964c95f4 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true} lago.ssh: DEBUG: start task:c85287ef-a0b5-4cea-ab1a-cfc1c4cc97d4:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:c85287ef-a0b5-4cea-ab1a-cfc1c4cc97d4:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 98aaf00c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 98aaf00c on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 98aaf00c on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true} lago.ssh: DEBUG: start task:c1aa3391-00ad-4c4a-a698-304aa8d8e598:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:c1aa3391-00ad-4c4a-a698-304aa8d8e598:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 9b10536e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9b10536e on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 9b10536e on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2955 (Sun Aug 26 23:30:50 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2957 (Sun Aug 26 23:30:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6d9e5c54", "local_conf_timestamp": 2957, "host-ts": 2955}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2952 (Sun Aug 26 23:30:46 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2953 (Sun Aug 26 23:30:47 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2260cf90", "local_conf_timestamp": 2953, "host-ts": 2952}, "global_maintenance": true} lago.ssh: DEBUG: start task:a7f2e7dd-ffdc-4265-a6b0-58c16e4d076f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:a7f2e7dd-ffdc-4265-a6b0-58c16e4d076f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 9d796f1e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9d796f1e on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 9d796f1e on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2955 (Sun Aug 26 23:30:50 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2957 (Sun Aug 26 23:30:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6d9e5c54", "local_conf_timestamp": 2957, "host-ts": 2955}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2952 (Sun Aug 26 23:30:46 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2953 (Sun Aug 26 23:30:47 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2260cf90", "local_conf_timestamp": 2953, "host-ts": 2952}, "global_maintenance": true} lago.ssh: DEBUG: start task:197b6456-4573-4fe5-a902-4d4135afd1c5:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:197b6456-4573-4fe5-a902-4d4135afd1c5:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 9fef209a on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9fef209a on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 9fef209a on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true} lago.ssh: DEBUG: start task:e449730a-24b3-4ba7-a32a-9841d681e8d7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:e449730a-24b3-4ba7-a32a-9841d681e8d7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running a251ea20 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a251ea20 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command a251ea20 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true} lago.ssh: DEBUG: start task:8fa960e8-d3d3-4de3-8673-7ccb2e8e3232:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:8fa960e8-d3d3-4de3-8673-7ccb2e8e3232:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running a4b99e3e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a4b99e3e on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command a4b99e3e on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true} lago.ssh: DEBUG: start task:df444f3d-0958-4232-bbe2-79c9b53d6df7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:df444f3d-0958-4232-bbe2-79c9b53d6df7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running a7386262 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a7386262 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command a7386262 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2975 (Sun Aug 26 23:31:10 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2976 (Sun Aug 26 23:31:11 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1cb13a70", "local_conf_timestamp": 2976, "host-ts": 2975}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2972 (Sun Aug 26 23:31:06 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2973 (Sun Aug 26 23:31:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "398afe7f", "local_conf_timestamp": 2973, "host-ts": 2972}, "global_maintenance": true} lago.ssh: DEBUG: start task:ccb637f6-e93c-474f-8822-99fa3e47e492:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:ccb637f6-e93c-474f-8822-99fa3e47e492:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2975 (Sun Aug 26 23:31:10 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2976 (Sun Aug 26 23:31:11 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1cb13a70", "local_conf_timestamp": 2976, "host-ts": 2975}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2972 (Sun Aug 26 23:31:06 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2973 (Sun Aug 26 23:31:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "398afe7f", "local_conf_timestamp": 2973, "host-ts": 2972}, "global_maintenance": true} lago.ssh: DEBUG: start task:3c133bbf-6ccf-4f97-a708-4902c159a68f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:3c133bbf-6ccf-4f97-a708-4902c159a68f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running ac14d18a on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ac14d18a on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command ac14d18a on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true} lago.ssh: DEBUG: start task:ff5cd16b-0c57-4df9-bbcc-9fd94b5fccb8:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:ff5cd16b-0c57-4df9-bbcc-9fd94b5fccb8:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true} lago.ssh: DEBUG: start task:7fcd4b38-7d15-4000-ad18-d363a3d3a7ab:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:7fcd4b38-7d15-4000-ad18-d363a3d3a7ab:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running b0c63368 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b0c63368 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command b0c63368 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true} lago.ssh: DEBUG: start task:3c20a1c0-ec61-41b6-b247-25ac0093a494:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:3c20a1c0-ec61-41b6-b247-25ac0093a494:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running b30f7738 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b30f7738 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command b30f7738 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7a1c6bf2", "local_conf_timestamp": 2993, "host-ts": 2992}, "global_maintenance": true} lago.ssh: DEBUG: start task:90657a8e-cbc1-4061-823a-c2e3d71635c3:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:90657a8e-cbc1-4061-823a-c2e3d71635c3:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running b555e324 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b555e324 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command b555e324 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f5d8fad1410> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) RuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 53, in restart_he_vm _wait_for_engine_health(host) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 185, in _wait_for_engine_health testlib.assert_true_within_long(lambda: any( File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) 'could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1",\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:a3c71d1c-cb12-4fef-9e6a-d5e0c1b07550:Get ssh client for lago-he-basic-iscsi-suite-master-host-1:\nlago.ssh: DEBUG: end task:a3c71d1c-cb12-4fef-9e6a-d5e0c1b07550:Get ssh client for lago-he-basic-iscsi-suite-master-host-1:\nlago.ssh: DEBUG: Running 43cfc44a on lago-he-basic-iscsi-suite-master-host-1: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 43cfc44a on lago-he-basic-iscsi-suite-master-host-1 returned with 0\nlago.ssh: DEBUG: Command 43cfc44a on lago-he-basic-iscsi-suite-master-host-1 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2805 (Sun Aug 26 23:28:20 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2806 (Sun Aug 26 23:28:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd22cf78", "local_conf_timestamp": 2806, "host-ts": 2805}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2810 (Sun Aug 26 23:28:24 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2812 (Sun Aug 26 23:28:26 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7f7e634d", "local_conf_timestamp": 2812, "host-ts": 2810}, "global_maintenance": true}\n\nroot: INFO: * Shutting down HE VM on host: lago-he-basic-iscsi-suite-master-host-0\nlago.ssh: DEBUG: start task:42afe970-0be5-40d6-834b-245d877c0d1a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:42afe970-0be5-40d6-834b-245d877c0d1a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 445276b0 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-shutdown\nlago.ssh: DEBUG: Command 445276b0 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be down...\nlago.ssh: DEBUG: start task:d92ec1bd-a571-4e86-b3a6-7bedd805567d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:d92ec1bd-a571-4e86-b3a6-7bedd805567d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 44f97960 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 44f97960 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 44f97960 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2805 (Sun Aug 26 23:28:20 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2806 (Sun Aug 26 23:28:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd22cf78", "local_conf_timestamp": 2806, "host-ts": 2805}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2800 (Sun Aug 26 23:28:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2801 (Sun Aug 26 23:28:15 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "063aa920", "local_conf_timestamp": 2801, "host-ts": 2800}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:a1e22b81-c1cf-4b25-a1f7-342286fbe81b:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:a1e22b81-c1cf-4b25-a1f7-342286fbe81b:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:f5948091-c497-4f3b-9761-04d42a2cc421:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:f5948091-c497-4f3b-9761-04d42a2cc421:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 4df162c6 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 4df162c6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 4df162c6 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8b462547-686e-4dbe-8bfb-91d4ba2931dd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:8b462547-686e-4dbe-8bfb-91d4ba2931dd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 504ec626 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 504ec626 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 504ec626 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:f2eef89a-f42e-4c79-9337-72571575f679:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:f2eef89a-f42e-4c79-9337-72571575f679:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 52a4916c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 52a4916c on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 52a4916c on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2835 (Sun Aug 26 23:28:50 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2837 (Sun Aug 26 23:28:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7a4710c6", "local_conf_timestamp": 2837, "host-ts": 2835}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2831 (Sun Aug 26 23:28:45 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2833 (Sun Aug 26 23:28:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fce69ba8", "local_conf_timestamp": 2833, "host-ts": 2831}, "global_maintenance": true}\n\nroot: INFO: * VM is down.\nroot: INFO: * Stopping services...\nlago.ssh: DEBUG: start task:2c25f4a6-f2ef-4cfa-822f-b929c4dfd509:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:2c25f4a6-f2ef-4cfa-822f-b929c4dfd509:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 5322b5b0 on lago-he-basic-iscsi-suite-master-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command 5322b5b0 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nroot: INFO: * Starting services...\nlago.ssh: DEBUG: start task:ee6b4609-b5ac-4f3c-a182-7ed44b8deeb6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ee6b4609-b5ac-4f3c-a182-7ed44b8deeb6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 5a5b1dd6 on lago-he-basic-iscsi-suite-master-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command 5a5b1dd6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nroot: INFO: * Waiting for agent to be ready...\nlago.ssh: DEBUG: start task:facd8e55-61ed-4fb7-b9fe-c3ad705d968a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:facd8e55-61ed-4fb7-b9fe-c3ad705d968a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:3eb14b7c-074b-4c11-a117-392f8b162c2e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:3eb14b7c-074b-4c11-a117-392f8b162c2e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 62a124ea on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 62a124ea on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 62a124ea on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:c78b3a53-f7bb-480e-839e-b57e9ca7eb4d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:c78b3a53-f7bb-480e-839e-b57e9ca7eb4d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:ca525d74-b2b6-4297-ad54-b044bc452568:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ca525d74-b2b6-4297-ad54-b044bc452568:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 66d48ffc on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 66d48ffc on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 66d48ffc on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:f871d6fb-e79f-477b-b5ca-99b939a4d9f7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:f871d6fb-e79f-477b-b5ca-99b939a4d9f7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 69018c26 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 69018c26 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 69018c26 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:b86743b4-04cf-4455-aba9-0bc61ad3815e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:b86743b4-04cf-4455-aba9-0bc61ad3815e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:8e87c593-e4ec-4db5-b533-eab312a2c1cb:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:8e87c593-e4ec-4db5-b533-eab312a2c1cb:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 6d468714 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 6d468714 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 6d468714 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:221b1e18-be90-427f-8a54-9291fe6f5959:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:221b1e18-be90-427f-8a54-9291fe6f5959:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 6f693f32 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 6f693f32 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 6f693f32 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:52fbd8f8-8d38-4465-9ecc-c00e381b7563:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:52fbd8f8-8d38-4465-9ecc-c00e381b7563:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 71a1db42 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 71a1db42 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 71a1db42 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:7a8d4a8c-12b5-4a68-b436-535b8d09f517:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:7a8d4a8c-12b5-4a68-b436-535b8d09f517:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:3de81b6a-badb-4222-8aca-128ebdfec400:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:3de81b6a-badb-4222-8aca-128ebdfec400:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 75dd4066 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 75dd4066 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 75dd4066 on lago-he-basic-iscsi-suite-master-host-0 output:\n \n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\n\n--== Host lago-he-basic-iscsi-suite-master-host-0.lago.local (id: 1) status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : True\nHostname : lago-he-basic-iscsi-suite-master-host-0.lago.local\nHost ID : 1\nEngine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}\nScore : 0\nstopped : False\nLocal maintenance : False\ncrc32 : 56f93df8\nlocal_conf_timestamp : 2896\nHost timestamp : 2895\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2895 (Sun Aug 26 23:29:50 2018)\n\thost-id=1\n\tscore=0\n\tvm_conf_refresh_time=2896 (Sun Aug 26 23:29:51 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=ReinitializeFSM\n\tstopped=False\n\n\n--== Host lago-he-basic-iscsi-suite-master-host-1 (id: 2) status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : False\nHostname : lago-he-basic-iscsi-suite-master-host-1\nHost ID : 2\nEngine status : unknown stale-data\nScore : 3400\nstopped : False\nLocal maintenance : False\ncrc32 : ffa35d6d\nlocal_conf_timestamp : 2892\nHost timestamp : 2891\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2891 (Sun Aug 26 23:29:45 2018)\n\thost-id=2\n\tscore=3400\n\tvm_conf_refresh_time=2892 (Sun Aug 26 23:29:46 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=GlobalMaintenance\n\tstopped=False\n\n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\nroot: INFO: * Agent is ready.\nroot: INFO: * Starting VM...\nlago.ssh: DEBUG: start task:0174ef46-3f09-4c57-b0cc-135fc059665a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:0174ef46-3f09-4c57-b0cc-135fc059665a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 764e856e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-start\nlago.ssh: DEBUG: Command 764e856e on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 764e856e on lago-he-basic-iscsi-suite-master-host-0 output:\n VM exists and is down, cleaning up and restarting\n\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be UP...\nlago.ssh: DEBUG: start task:30ee44ce-7f0b-4b16-ba1b-7b6964891efd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:30ee44ce-7f0b-4b16-ba1b-7b6964891efd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 77b7ccda on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 77b7ccda on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 77b7ccda on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2895 (Sun Aug 26 23:29:50 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=2896 (Sun Aug 26 23:29:51 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=ReinitializeFSM\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "56f93df8", "local_conf_timestamp": 2896, "host-ts": 2895}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2891 (Sun Aug 26 23:29:45 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2892 (Sun Aug 26 23:29:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ffa35d6d", "local_conf_timestamp": 2892, "host-ts": 2891}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ef26fadc-fd32-4afb-9a06-07e13a3e556f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ef26fadc-fd32-4afb-9a06-07e13a3e556f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2905 (Sun Aug 26 23:30:00 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2906 (Sun Aug 26 23:30:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9fce1b92", "local_conf_timestamp": 2906, "host-ts": 2905}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2901 (Sun Aug 26 23:29:55 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2902 (Sun Aug 26 23:29:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "68110bea", "local_conf_timestamp": 2902, "host-ts": 2901}, "global_maintenance": true}\n\nroot: INFO: * VM is UP.\nroot: INFO: * Waiting for engine to start...\nlago.ssh: DEBUG: start task:22f9ce85-5236-43de-956e-60eedabdee44:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:22f9ce85-5236-43de-956e-60eedabdee44:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2905 (Sun Aug 26 23:30:00 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2906 (Sun Aug 26 23:30:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9fce1b92", "local_conf_timestamp": 2906, "host-ts": 2905}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2901 (Sun Aug 26 23:29:55 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2902 (Sun Aug 26 23:29:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "68110bea", "local_conf_timestamp": 2902, "host-ts": 2901}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:10d11112-af95-4062-82b5-90a12379d806:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:10d11112-af95-4062-82b5-90a12379d806:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 85234b10 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 85234b10 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 85234b10 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2915 (Sun Aug 26 23:30:10 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2916 (Sun Aug 26 23:30:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69a0d34f", "local_conf_timestamp": 2916, "host-ts": 2915}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2911 (Sun Aug 26 23:30:05 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2913 (Sun Aug 26 23:30:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ff5ba0ab", "local_conf_timestamp": 2913, "host-ts": 2911}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8600ff29-4795-4922-9bb5-1ea8cb47aff6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:8600ff29-4795-4922-9bb5-1ea8cb47aff6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 8778cab6 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 8778cab6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 8778cab6 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2915 (Sun Aug 26 23:30:10 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2916 (Sun Aug 26 23:30:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69a0d34f", "local_conf_timestamp": 2916, "host-ts": 2915}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:6251e2ec-36ea-42cd-a7cf-018d3efaf73a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:6251e2ec-36ea-42cd-a7cf-018d3efaf73a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 89e10548 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 89e10548 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 89e10548 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2925 (Sun Aug 26 23:30:20 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2927 (Sun Aug 26 23:30:22 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "df65bcff", "local_conf_timestamp": 2927, "host-ts": 2925}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:5671a60c-09fc-4f60-97a2-270f9f5b4ca1:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:5671a60c-09fc-4f60-97a2-270f9f5b4ca1:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2925 (Sun Aug 26 23:30:20 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2927 (Sun Aug 26 23:30:22 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "df65bcff", "local_conf_timestamp": 2927, "host-ts": 2925}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ca12f761-8a01-4613-8217-3848165adb82:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ca12f761-8a01-4613-8217-3848165adb82:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 8efac10e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 8efac10e on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 8efac10e on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2935 (Sun Aug 26 23:30:30 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2937 (Sun Aug 26 23:30:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "41a00e65", "local_conf_timestamp": 2937, "host-ts": 2935}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2932 (Sun Aug 26 23:30:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2934 (Sun Aug 26 23:30:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "903a0802", "local_conf_timestamp": 2934, "host-ts": 2932}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:2ce2a039-af5f-47e2-b0c8-650b8ca40276:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:2ce2a039-af5f-47e2-b0c8-650b8ca40276:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2935 (Sun Aug 26 23:30:30 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2937 (Sun Aug 26 23:30:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "41a00e65", "local_conf_timestamp": 2937, "host-ts": 2935}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2932 (Sun Aug 26 23:30:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2934 (Sun Aug 26 23:30:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "903a0802", "local_conf_timestamp": 2934, "host-ts": 2932}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:50565f8c-6b24-4ae7-9eba-bbf9b31b3867:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:50565f8c-6b24-4ae7-9eba-bbf9b31b3867:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 93ca2012 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 93ca2012 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 93ca2012 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8de05a79-ab0e-4659-bc5c-03ab090af777:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:8de05a79-ab0e-4659-bc5c-03ab090af777:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 964c95f4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 964c95f4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 964c95f4 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:c85287ef-a0b5-4cea-ab1a-cfc1c4cc97d4:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:c85287ef-a0b5-4cea-ab1a-cfc1c4cc97d4:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 98aaf00c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 98aaf00c on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 98aaf00c on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:c1aa3391-00ad-4c4a-a698-304aa8d8e598:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:c1aa3391-00ad-4c4a-a698-304aa8d8e598:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 9b10536e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9b10536e on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 9b10536e on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2955 (Sun Aug 26 23:30:50 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2957 (Sun Aug 26 23:30:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6d9e5c54", "local_conf_timestamp": 2957, "host-ts": 2955}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2952 (Sun Aug 26 23:30:46 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2953 (Sun Aug 26 23:30:47 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2260cf90", "local_conf_timestamp": 2953, "host-ts": 2952}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:a7f2e7dd-ffdc-4265-a6b0-58c16e4d076f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:a7f2e7dd-ffdc-4265-a6b0-58c16e4d076f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 9d796f1e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9d796f1e on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 9d796f1e on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2955 (Sun Aug 26 23:30:50 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2957 (Sun Aug 26 23:30:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6d9e5c54", "local_conf_timestamp": 2957, "host-ts": 2955}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2952 (Sun Aug 26 23:30:46 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2953 (Sun Aug 26 23:30:47 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2260cf90", "local_conf_timestamp": 2953, "host-ts": 2952}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:197b6456-4573-4fe5-a902-4d4135afd1c5:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:197b6456-4573-4fe5-a902-4d4135afd1c5:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 9fef209a on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9fef209a on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 9fef209a on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:e449730a-24b3-4ba7-a32a-9841d681e8d7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:e449730a-24b3-4ba7-a32a-9841d681e8d7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running a251ea20 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a251ea20 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a251ea20 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8fa960e8-d3d3-4de3-8673-7ccb2e8e3232:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:8fa960e8-d3d3-4de3-8673-7ccb2e8e3232:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running a4b99e3e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a4b99e3e on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a4b99e3e on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:df444f3d-0958-4232-bbe2-79c9b53d6df7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:df444f3d-0958-4232-bbe2-79c9b53d6df7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running a7386262 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a7386262 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a7386262 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2975 (Sun Aug 26 23:31:10 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2976 (Sun Aug 26 23:31:11 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1cb13a70", "local_conf_timestamp": 2976, "host-ts": 2975}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2972 (Sun Aug 26 23:31:06 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2973 (Sun Aug 26 23:31:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "398afe7f", "local_conf_timestamp": 2973, "host-ts": 2972}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ccb637f6-e93c-474f-8822-99fa3e47e492:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ccb637f6-e93c-474f-8822-99fa3e47e492:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2975 (Sun Aug 26 23:31:10 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2976 (Sun Aug 26 23:31:11 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1cb13a70", "local_conf_timestamp": 2976, "host-ts": 2975}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2972 (Sun Aug 26 23:31:06 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2973 (Sun Aug 26 23:31:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "398afe7f", "local_conf_timestamp": 2973, "host-ts": 2972}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:3c133bbf-6ccf-4f97-a708-4902c159a68f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:3c133bbf-6ccf-4f97-a708-4902c159a68f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running ac14d18a on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ac14d18a on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ac14d18a on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ff5cd16b-0c57-4df9-bbcc-9fd94b5fccb8:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ff5cd16b-0c57-4df9-bbcc-9fd94b5fccb8:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:7fcd4b38-7d15-4000-ad18-d363a3d3a7ab:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:7fcd4b38-7d15-4000-ad18-d363a3d3a7ab:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running b0c63368 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b0c63368 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command b0c63368 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:3c20a1c0-ec61-41b6-b247-25ac0093a494:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:3c20a1c0-ec61-41b6-b247-25ac0093a494:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running b30f7738 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b30f7738 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command b30f7738 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7a1c6bf2", "local_conf_timestamp": 2993, "host-ts": 2992}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:90657a8e-cbc1-4061-823a-c2e3d71635c3:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:90657a8e-cbc1-4061-823a-c2e3d71635c3:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running b555e324 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b555e324 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command b555e324 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1",\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f5d8fad1410>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda>\n for k, v in _get_he_status(host).items()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status\n raise RuntimeError(\'could not parse JSON: %s\' % ret.out)\nRuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1",\n--------------------- >> end captured logging << ---------------------'

1 2

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #1884
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18

27 Aug '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 3afc9ba57755a1b2574f7a177184eb66d5f6a90d (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 3afc9ba57755a1b2574f7a177184eb66d5f6a90d Commit message: "ovirt-release: drop non working jobs" > git rev-list --no-walk 3afc9ba57755a1b2574f7a177184eb66d5f6a90d # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins1863032023757664663.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7' Build step 'Execute shell' marked build as failure

1 1

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build # 639 - Failure!
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18

27 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/639/ Build Number: 639 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #639 [Roy Golan] Introduce openshift-on-ovirt ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.snapshot_merge Error Message: status: 409 reason: Conflict detail: Cannot create Snapshot. Snapshot is currently being created for VM vm0. Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 69, in wrapper return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/004_basic_sanity.py", line 194, in snapshot_merge api.vms.get(VM0_NAME).snapshots.add(dead_snap2_params) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 34511, in add headers={"Correlation-Id":correlation_id, "Expect":expect} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add return self.request('POST', url, body, headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) RequestError: status: 409 reason: Conflict detail: Cannot create Snapshot. Snapshot is currently being created for VM vm0.

1 1

[CQ]: 93756,2 (imgbased) failed "ovirt-4.2" system tests
by oVirt Jenkins 27 Aug '18

27 Aug '18

Change 93756,2 (imgbased) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/93756/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2986/

1 0

[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-4.2 - Build # 255 - Failure!
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18

27 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/255/ Build Number: 255 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #255 [Roy Golan] Introduce openshift-on-ovirt ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 012_local_maintenance_sdk.local_maintenance Error Message: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/012_local_maintenance_sdk.py", line 45, in local_maintenance vm_service = test_utils.get_vm_service(engine, VM_HE_NAME) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/test_utils/__init__.py", line 71, in __call__ value = self.func(*args) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/test_utils/__init__.py", line 101, in get_vm_service vm = vms_service.list(search='name={}'.format(vm_name))[0] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 34154, in list return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type raise Error(msg) "The response content type 'text/html; charset=iso-8859-1' isn't the expected XML\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\n--------------------- >> end captured logging << ---------------------"

1 1

[oVirt Jenkins] ovirt-appliance_master_build-artifacts-fc28-x86_64 - Build # 40 - Failure!
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18

27 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-fc28-x8… Build: http://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-fc28-x8… Build Number: 40 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #40 [Yuval Turgeman] automation: remove build-artifacts* from stdci-v2 ----------------- Failed Tests: ----------------- No tests ran.

1 1

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #1880
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18

27 Aug '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 3afc9ba57755a1b2574f7a177184eb66d5f6a90d (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 3afc9ba57755a1b2574f7a177184eb66d5f6a90d Commit message: "ovirt-release: drop non working jobs" > git rev-list --no-walk 3afc9ba57755a1b2574f7a177184eb66d5f6a90d # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins5224362443311299032.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7' Build step 'Execute shell' marked build as failure

1 3

[oVirt Jenkins] ovirt-system-tests_he-basic-suite-4.2 - Build # 510 - Failure!
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18

27 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.2/510/ Build Number: 510 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #510 [Roy Golan] Introduce openshift-on-ovirt ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 006_network_by_label.assign_hosts_network_label Error Message: status: 409 reason: Conflict detail: Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later. -------------------- >> begin captured logging << -------------------- lago.utils: ERROR: Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/006_network_by_label.py", line 57, in _assign_host_network_label host_nic=nic File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 16231, in add headers={"Correlation-Id":correlation_id, "Expect":expect} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add return self.request('POST', url, body, headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) RequestError: status: 409 reason: Conflict detail: Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later. --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 69, in wrapper return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/006_network_by_label.py", line 65, in assign_hosts_network_label nt.assert_true(all(vt.join_all())) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/006_network_by_label.py", line 57, in _assign_host_network_label host_nic=nic File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 16231, in add headers={"Correlation-Id":correlation_id, "Expect":expect} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add return self.request('POST', url, body, headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) '\r\nstatus: 409\r\nreason: Conflict\r\ndetail: Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later.\n-------------------- >> begin captured logging << --------------------\nlago.utils: ERROR: Error while running thread\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue\n queue.put({\'return\': func()})\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/006_network_by_label.py", line 57, in _assign_host_network_label\n host_nic=nic\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 16231, in add\n headers={"Correlation-Id":correlation_id, "Expect":expect}\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add\n return self.request(\'POST\', url, body, headers, cls=cls)\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request\n persistent_auth=self.__persistent_auth\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request\n persistent_auth)\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request\n raise errors.RequestError(response_code, response_reason, response_body)\nRequestError: \r\nstatus: 409\r\nreason: Conflict\r\ndetail: Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later.\n--------------------- >> end captured logging << ---------------------'

1 2

Build failed in Jenkins: system-sync_mirrors-epel-el6-x86_64 #1802
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18

27 Aug '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/1802/displ…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 3afc9ba57755a1b2574f7a177184eb66d5f6a90d (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 3afc9ba57755a1b2574f7a177184eb66d5f6a90d Commit message: "ovirt-release: drop non working jobs" > git rev-list --no-walk 3afc9ba57755a1b2574f7a177184eb66d5f6a90d # timeout=10 [system-sync_mirrors-epel-el6-x86_64] $ /bin/bash -xe /tmp/jenkins4293223092552671319.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror epel-el6 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Resyncing repo: epel-el6 Syncing yum repo epel-el6 for arch: x86_64 epel-el6/group | 303 kB 00:00 tklib-0.5-5.el6.noarch.rpm 0% [ ] 0.0 B/s | 0 B --:-- ETA tklib-0.5-5.el6.noarch.rpm | 416 kB 00:02 Generating yum repo metadata for: epel-el6 Worker exited with non-zero value: -9. Fatal. Worker exited with non-zero value: -9. Fatal. Spawning worker 0 with 1519 pkgs Spawning worker 1 with 1519 pkgs Spawning worker 2 with 1519 pkgs Spawning worker 3 with 1518 pkgs Spawning worker 4 with 1518 pkgs Spawning worker 5 with 1518 pkgs Spawning worker 6 with 1518 pkgs Spawning worker 7 with 1518 pkgs Build step 'Execute shell' marked build as failure

1 1

oVirt infra daily report - unstable production jobs - 564
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18

27 Aug '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/564//artifact/exported-a… Cheers, Jenkins

1 0

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build # 637 - Failure!
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18

27 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/637/ Build Number: 637 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #637 [Roy Golan] Introduce openshift-on-ovirt ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.add_hosts Error Message: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML -------------------- >> begin captured logging << -------------------- ovirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7f031a4cfed8> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4 host_obj = host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 36676, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 228, in add_hosts add_hosts_4(prefix) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 275, in add_hosts_4 testlib.assert_true_within(_host_is_up_4, timeout=15*60) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 278, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4 host_obj = host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 36676, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type raise Error(msg) 'The response content type \'text/html; charset=iso-8859-1\' isn\'t the expected XML\n-------------------- >> begin captured logging << --------------------\novirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7f031a4cfed8>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4\n host_obj = host_service.get()\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 36676, in get\n return self._internal_get(headers, query, wait)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get\n return future.wait() if wait else future\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait\n return self._code(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback\n self._check_fault(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault\n body = self._internal_read_body(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body\n self._connection.check_xml_content_type(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type\n response.headers\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type\n raise Error(msg)\nError: The response content type \'text/html; charset=iso-8859-1\' isn\'t the expected XML\n--------------------- >> end captured logging << ---------------------'

1 1

[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-master - Build # 318 - Failure!
by jenkins＠jenkins.phx.ovirt.org 26 Aug '18

26 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/318/ Build Number: 318 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #318 [Roy Golan] Introduce openshift-on-ovirt ----------------- Failed Tests: ----------------- No tests ran.

1 2

[oVirt Jenkins] ovirt-system-tests_he-basic-iscsi-suite-master - Build # 462 - Failure!
by jenkins＠jenkins.phx.ovirt.org 26 Aug '18

26 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master… Build Number: 462 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #462 [Roy Golan] Introduce openshift-on-ovirt ----------------- Failed Tests: ----------------- No tests ran.

1 1